Re: [opensuse] optimize dd

On Wed, Mar 19, 2008 at 10:09 AM, Otto Rodusek (AP-SGP)
<otto@xxxxxxxxxxxxxx> wrote:
Greg Freemyer wrote:
> On Wed, Mar 19, 2008 at 8:10 AM, Per Jessen <per@xxxxxxxxxxxx> wrote:
>
>> Otto Rodusek (AP-SGP) wrote:
>>
>> > Is there an optimum 'bs' parameter to the dd command. For example if
>> > you want to clone 2 drives (dd if=/dev/hda of=/dev/hdb bs=??) is there
>> > a better value to bs that will finish the process faster? Will a
>> > larger value like bs=8257536 or a value like bs=32256 be better? Is it
>> > better to have a value that is a multiple of sectors (ie 63 sectors @
>> > 512 bytes = 32256)? Thanks for any comments. Otto.
>>
>> Generally speaking - bigger blocksize, better performance. It will
>> depend on your drives though, so why not just try it out? Just do some
>> plain copies of fixed sized files and measure wall clock time.
>>
>
> dd if=/dev/hdx of=/dev/hdy
> is not the same thing as
> dd if=file of=file
>
> Files imply filesystem which brings into play a whole new set of
> behavior. With files on a filesystem you will see a huge difference
> between bs=512 and bs=4K as an example.
> Hi,


> Is there an optimum 'bs' parameter to the dd command. For example if you want to clone 2 drives (dd if=/dev/hda of=/dev/hdb bs=??) is there a better value to bs that will finish the process faster? Will a larger value like bs=8257536 or a value like bs=32256 be better? Is it better to have a value that is a multiple of sectors (ie 63 sectors @ 512 bytes = 32256)? Thanks for any comments. Otto.
>

With a clone, I don't think you will see a readily observable
> difference between those 2 values.
>
> Greg
>
Hi Greg,

Thanks for the info - I was actually hoping for a tried and tested
answer. I will probably do a:

time dd if=/dev/hda of=/dev/hdb bs=??


Otto,

It has been a while, but I developed my 4KB choice about 5 years ago.
(SUSE 8.2 I think). Obviously I was using IDE drives, not sata etc.

I did a number of tests back then and did not find any reason to go
past 4 KB. But newer hardware may have changed all that. Worth
testing again I suppose.

If you do make sure you include flushing the OS cache.

Something like:
sync; sync; time (dd if=/dev/sda of=/dev/sdb bs=4k count=100; sync; sync)

More important than BS, was hardware issues. As I said, we do a lot
of disk cloning so we have configured at least a dozen machines in our
lab to be as efficient at this process as possible.

Things I learned from back then. All IDE related only. I have NOT
redone any of these tests for sata controllers / drives.

1) Never use a master and slave on the same IDE channel.. Extremely slow.

2) Going from master to master on the same Motherboard IDE controller
is better, but it is still significantly slower than having a PCI IDE
controller installed and copying from the MB IDE controller to the PCI
IDE controller.

3) Surprising to me was that the fastest solution of all was to use a
dual channel PCI card and go between the 2 channels. (~$35)

4) A DMA-133 drive is really faster than a DMA-100 drive for cloning.
(Yes, even with 4KB blocksize. This basically means your maxing out
the hardware with 4KB bs, so trying to optimize BS is not going to do
much.)

5) Laptop drives are slow as sin. Not much you can do about that if
they are what you need to work with.

6) USB is very slow. Much slower than you would expect from the spec.
I suspect it disables interrupts way too much and prevents parrallel
read write activity to the IDE drive.

7) Firewire is not much better in some situations. (We don't do very
much with firewire.)

And in the world of hearsay.

Sata-150 is really fast. We see cloning speeds of up to 4 GB/min.
(with 4 KB bs). Unfortunately I have not really paid attention to the
architectures that produce this. Seems like it has been using the
Sata ports on the MB of new generation MBs. We don't get that sort of
speed out of PCI Sata controllers.

Sata-300 is no faster at all. ie. Sata-150 seems to keep up with the
physical drive, so the sata cable is not the bottleneck. No advantage
with a simple controller <--> drive config to use sata-300. (If you
have a PMP in the middle, then sata-300 is helpful. We have not used
a PMP config yet.)

The fastest solution is probably PCI-express controllers. We have not
done any experiments with that yet either.

Greg
--
Greg Freemyer
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
--
To unsubscribe, e-mail: opensuse+unsubscribe@xxxxxxxxxxxx
For additional commands, e-mail: opensuse+help@xxxxxxxxxxxx

Relevant Pages

  • Re: Install Sata drivers
    ... >>> beacause it already includes all the required sata, raid, etc drivers. ... >>the purposes of allowing WinXP to install without pressing F6. ... > connected via IDE -- i.e., there's no necessity for these drives to ...
    (alt.comp.periphs.mainboard.asus)
  • Re: Windows Appears to Recognize a SATA Drive as an IDE Drive
    ... The results for your P-ATA drives are reasonable for SUSTAINED (not BURST ... IDE Channel>>Advanced Settings? ... The SATA transfer rates are abssurdly low, and even the IDE speeds are far ... NERO IMAGEDRIVE SCSI Controller ...
    (microsoft.public.windowsxp.configuration_manage)
  • Re: PATA HDD and only 2 IDE connectors
    ... It has 2 IDE connectors and 4 SATA connectors. ... the drives, I wouldn't even thing of going there. ...
    (comp.sys.ibm.pc.hardware.chips)
  • Re: Is a Seagate 300Gb Ultra ATA/100 IDE int hd compatible with my pc?
    ... If you have the drives you're supposed to, I don't know why you'd see ... I thought that I had purchased only an IDE int hd. ... Primary IDE Channel [Controller] ... VIA USB Enhanced Host Controller ...
    (microsoft.public.windowsxp.hardware)
  • Re: hard drive install
    ... The IDE power cables, which there are 3 connectors for, have ... > from the 2 CD drives to the bottom of the case where the HD will be. ... >> since I've installed HDs before, but I guess I've only heard of SATA ...
    (microsoft.public.windowsxp.hardware)