FTP hangs

From: Dave Sill (MaxFreedom_at_sws5.ornl.gov)
Date: 08/24/05


Date: Wed, 24 Aug 2005 16:03:37 -0400

I've got an RHEL 3 server running a vsftpd FTP daemon. We're seeing
occasional temporary hangs when certain systems retrieve certain
files. For example, from a particular Solaris client:

----
root@client# ftp server
Connected to server.
220 NOTE: Please login as user ftp
Name (server:root): ftp
331 Please specify the password.
Password:
230 Login successful.
ftp> bin
200 Switching to Binary mode.
ftp> prom
Interactive mode off.
ftp> cd test1
250 Directory successfully changed.
ftp> mget *
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for sgpmfrsrE3.b1.20040602.000000.cdf.v0 (816156 bytes).
226 File send OK.
local: sgpmfrsrE3.b1.20040602.000000.cdf.v0 remote: sgpmfrsrE3.b1.20040602.000000.cdf.v0
816156 bytes received in 0.48 seconds (1665.80 Kbytes/s)
200 PORT command successful. Consider using PASV.
  <delay of several minutes here>
150 Opening BINARY mode data connection for sgpmfrsrE3.b1.20040602.180300.cdf.v0 (297204 bytes).
226 File send OK.
local: sgpmfrsrE3.b1.20040602.180300.cdf.v0 remote: sgpmfrsrE3.b1.20040602.180300.cdf.v0
297204 bytes received in 93 seconds (3.14 Kbytes/s)
ftp> by
221 Goodbye.
root@client# 
----
Once a session hits the delay point indicated above, further attempts
to ftp from the client to the server hang:
root@client# ftp server
Connected to server.
  <delay of here, sometimes permanent>
Attempts to connect from other clients during this lull experience no
delays.
Things we've discovered:
- grabbing the files individually with get in the same order produces
  the same hang
- grabbing the files in the reverse order doesn't hang
- renaming the files has no effect
- replacing the file contents with /dev/random or /dev/zero contents
  of the same size has no effect
- retrieving the same files from another client running the same OS
  release on the same hardware doesn't hang
- serving the same files from another server running the same OS on
  the same hardware doesn't hang
- files that some offsite user is having trouble retrieving don't
  exhibit the same problems internally, even from the known
  problematic client
- the problem is highly repeatable for a given client system and set
  of files, but rarely even these cases proceed normally
Has anyone ever seen anything like this? Any ideas what could be
causing it or how to debug it?
-- 
Dave Sill                     Oak Ridge National Lab, Workstation Support
Author, The qmail Handbook                <http://web.infoave.net/~dsill>
<http://lifewithqmail.org/>: Almost everything you always wanted to know.