Re: NFS and kernel cache

Once upon a time, Les Mikesell <lesmikesell@xxxxxxxxx> said:
On Thu, 2006-12-21 at 08:14 -0600, Chris Adams wrote:
The fact that Linux stops sending on the network sometimes and stops
reading the hard drive other times points directly to how the kernel is
caching writes to NFS (but I can't tell if it is the filesystem layer or
the network stack).

I think it is access to the physical hard drive, not particularly
NFS that is your bottleneck and it is related to normal buffered
writes. You'll see about the same pattern if you dd between two
local hard drives. If your nfs mount has the sync option it might
make things worse.

No, because on both ends, the hard drive goes idle. If I test the speed
of the hard drives (using bonnie++), I get higher rates. I did some
more testing.

a) If I read from the client hard drive with ntfsclone and write to an
NFS filesystem (across the gigE link), I get about 200Mbps.

b) If I bypass NFS and just use netcat on each end to send/receive data
(still reading from the client hard drive with ntfsclone and writing
to the server hard drive by just redirecting the output of nc), I get
about 278Mbps.

c) If I run the same test on the client but just dump to /dev/null on
the client (no network involved), the hard drive LED is on
continuously and I get about 312Mbps.

d) If I dd from /dev/zero on the client (using bs=1024), pipe to netcat,
and write to /dev/null on the server, I get about 343Mbps.

So, it would appear that NFS is my biggest choke point, but there is
still some buffering issue between reading from the hard drive and
writing to the network.

I tried doing the dd test on another client system, and I got up to
894Mbps (using jumbo frames gets it up to about 971Mbps), so I guess
part of my problem is the network interface in the first client
(Broadcom NetXtreme BCM5751M); the server is a Broadcom NetXtreme
BCM5702X and the second client is a Broadcom NetXtreme BCM5704.

--
Chris Adams <cmadams@xxxxxxxxxx>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

--
fedora-list mailing list
fedora-list@xxxxxxxxxx
To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list

Relevant Pages

  • Re: 100Mbps Ethernet Speed/Efficiency
    ... I took the hard drives out of the picture. ... I copied from the first system's RAM disk over ... Granted, NFS is in the ...
    (Fedora)
  • Re: NFS and kernel cache
    ... caching writes to NFS (but I can't tell if it is the filesystem layer or ... NFS that is your bottleneck and it is related to normal buffered ... local hard drives. ...
    (Fedora)
  • Re: 100Mbps Ethernet Speed/Efficiency
    ... I copied from the first system's RAM disk over ... Granted, NFS is in the ... > picture, but it would be using hard drives, too. ...
    (Fedora)
  • Errors writing large files via NFS
    ... files larger than a certain size to a NFS server. ... client systems, although the definition of "too large" varies. ... network paths involved, I'm pretty sure we're not seeing a network problem. ...
    (Tru64-UNIX-Managers)
  • Re: Internal Hacker
    ... You could check the local hard drives, ... search the hard drives across the network, or if you have a proxy server, ... which case the logs still wouldn't tell you who was using that account. ... across the network, install all patches, secure your systems per the ...
    (microsoft.public.security)