Re: Where is the performance bottleneck?

From: Vojtech Pavlik (vojtech_at_suse.cz)
Date: 08/31/05

  • Next message: Gaurav Dhiman: "Re: Device Drivers" 
    Date:	Wed, 31 Aug 2005 09:11:26 +0200
To: Holger Kiehl <Holger.Kiehl@dwd.de>

    On Tue, Aug 30, 2005 at 08:06:21PM +0000, Holger Kiehl wrote:
    > >>How does one determine the PCI-X bus speed?
    > >
    > >Usually only the card (in your case the Symbios SCSI controller) can
    > >tell. If it does, it'll be most likely in 'dmesg'.
    > >
    > There is nothing in dmesg:
    >
    > Fusion MPT base driver 3.01.20
    > Copyright (c) 1999-2004 LSI Logic Corporation
    > ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 24 (level, low) -> IRQ 217
    > mptbase: Initiating ioc0 bringup
    > ioc0: 53C1030: Capabilities={Initiator,Target}
    > ACPI: PCI Interrupt 0000:02:04.1[B] -> GSI 25 (level, low) -> IRQ 225
    > mptbase: Initiating ioc1 bringup
    > ioc1: 53C1030: Capabilities={Initiator,Target}
    > Fusion MPT SCSI Host driver 3.01.20
    >
    > >To find where the bottleneck is, I'd suggest trying without the
    > >filesystem at all, and just filling a large part of the block device
    > >using the 'dd' command.
    > >
    > >Also, trying without the RAID, and just running 4 (and 8) concurrent
    > >dd's to the separate drives could show whether it's the RAID that's
    > >slowing things down.
    > >
    > Ok, I did run the following dd command in different combinations:
    >
    > dd if=/dev/zero of=/dev/sd?1 bs=4k count=5000000

    I think a bs of 4k is way too small and will cause huge CPU overhead.
    Can you try with something like 4M? Also, you can use /dev/full to avoid
    the pre-zeroing.

    > Here the results:
    >
    > Each disk alone
    > /dev/sdc1 59.094636 MB/s
    > /dev/sdd1 58.686592 MB/s
    > /dev/sde1 55.282807 MB/s
    > /dev/sdf1 62.271240 MB/s
    > /dev/sdg1 60.872891 MB/s
    > /dev/sdh1 62.252781 MB/s
    > /dev/sdi1 59.145637 MB/s
    > /dev/sdj1 60.921119 MB/s

    > All 8 disks in parallel
    > /dev/sdc1 24.120545 MB/s
    > /dev/sdd1 24.419801 MB/s
    > /dev/sde1 24.296588 MB/s
    > /dev/sdf1 25.609548 MB/s
    > /dev/sdg1 24.572617 MB/s
    > /dev/sdh1 25.552590 MB/s
    > /dev/sdi1 24.575616 MB/s
    > /dev/sdj1 25.124165 MB/s

    You're saturating some bus. It almost looks like it's the PCI-X,
    although that should be able to deliver up (if running at the full speed
    of AMD8132) up to 1GB/sec, so it SHOULD not be an issue.

    > So from these results, I may assume that md is not the cause of the problem.
    >
    > What comes as a big surprise is that I loose 25% performance with only
    > two disks and each hanging on its own channel!
    >
    > Is this normal? I wonder if other people have the same problem with
    > other controllers or the same.

    No, I don't think this is OK.

    > What can I do next to find out if this is a kernel, driver or hardware
    > problem?
     
    You need to find where the bottleneck is, by removing one possible
    bottleneck at a time in your test. 

    -- 
Vojtech Pavlik
SuSE Labs, SuSE CR
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
  • Next message: Gaurav Dhiman: "Re: Device Drivers"

    Relevant Pages