Re: NFS mount hangs when using one specific IP

From: Michael Ritzert (ritzert_at_t-online.de)
Date: 05/19/05 

Date: Thu, 19 May 2005 20:58:15 +0200

Menno Duursma wrote:
> On Thu, 19 May 2005 15:29:43 +0200, Michael Ritzert wrote:
>
>> Forgot to mention: When I configure another computer on the IP in
>> question, it also doesn't work. mount just hangs and can't be killed.
>> So it's really IP specific.
>
> Sounds like a portmapper problem. How are hosts.{allow,deny} configured
> (remember for the rpc.portmap you can only use IP adress lists or "ALL".)

hosts.allow is empty, hosts.deny contains only
http-rman : ALL EXCEPT LOCAL

I grepped all of /etc for our IP address prefix and the IP in question and
found nothing of interest.

> I'd check both client and server settings, and stuff like:
>
> rpcinfo
> nfsstat
> showmount

run on the client:
# rpcinfo -t server nfs
program 100003 version 2 ready and waiting
program 100003 version 3 ready and waiting
# rpcinfo -u server nfs
program 100003 version 2 ready and waiting
program 100003 version 3 ready and waiting

# rpcinfo -t server mount
program 100005 version 1 ready and waiting
program 100005 version 2 ready and waiting
program 100005 version 3 ready and waiting
# rpcinfo -u server mount
program 100005 version 1 ready and waiting
program 100005 version 2 ready and waiting
program 100005 version 3 ready and waiting

# rpcinfo -p server
    100000 2 tcp 111 portmapper
    100000 2 udp 111 portmapper
    100003 2 udp 2049 nfs
    100003 3 udp 2049 nfs
    100227 3 udp 2049 nfs_acl
    100003 2 tcp 2049 nfs
    100003 3 tcp 2049 nfs
    100227 3 tcp 2049 nfs_acl
    100021 1 udp 32771 nlockmgr
    100021 3 udp 32771 nlockmgr
    100021 4 udp 32771 nlockmgr
    100024 1 udp 32771 status
    100021 1 tcp 32776 nlockmgr
    100021 3 tcp 32776 nlockmgr
    100021 4 tcp 32776 nlockmgr
    100024 1 tcp 32776 status
    100005 1 udp 1010 mountd
    100005 1 tcp 1013 mountd
    100005 2 udp 1010 mountd
    100005 2 tcp 1013 mountd
    100005 3 udp 1010 mountd
    100005 3 tcp 1013 mountd

# showmount -e server
Export list for server:
[...]
/home client
[...]

after the mount:
# nfsstat -c
Client rpc stats:
calls retrans authrefrsh
1 0 0
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
read write create mkdir symlink mknod
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
fsstat fsinfo pathconf commit
0 0% 1 100% 0 0% 0 0%

In the meantime, I installed tcpdump and let it trace the traffic between
the two hosts. In my analysis I can see that the server receives an FSINFO
call, but never answers. That fits well to the nfsstat output above.
When I perform a mount from another host, the initial sequence is the same
(down to the relative packet numbers), but the server sends the expected
reply to the FSINFO call.

BTW: Should the incorrect checksums sent by the server bother me? Or does
the NIC correct these?

Another idea I have is to set up a second NFS server and see if I can mount
a directory from this one.

Michael

TCP Dump: (stripped all the ACK,SYN,FIN stuff)

No. Time Source Destination Protocol
Info
      4 0.000148 1.2.3.210 1.2.3.111 Portmap V2
GETPORT Call (Reply In 6) NFS(100003) V:3 TCP

Frame 4 (126 bytes on wire, 126 bytes captured)
Ethernet II, Src: 12:34:56:78:9a:bc, Dst: fe:dc:ba:98:76:54
Internet Protocol, Src Addr: 1.2.3.210 (1.2.3.210), Dst Addr: 1.2.3.111
(1.2.3.111)
Transmission Control Protocol, Src Port: 32781 (32781), Dst Port: sunrpc
(111), Seq: 1, Ack: 1, Len: 60
    Source port: 32781 (32781)
    Destination port: sunrpc (111)
    Sequence number: 1 (relative sequence number)
    Next sequence number: 61 (relative sequence number)
    Acknowledgement number: 1 (relative ack number)
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
    Window size: 5840 (scaled)
    Checksum: 0xc51f (correct)
    Options: (12 bytes)
Remote Procedure Call, Type:Call XID:0x295f366c
Portmap GETPORT Call NFS(100003) Version:3 TCP

No. Time Source Destination Protocol
Info
      6 0.000356 1.2.3.111 1.2.3.210 Portmap V2
GETPORT Reply (Call In 4) Port:2049

Frame 6 (98 bytes on wire, 98 bytes captured)
Ethernet II, Src: fe:dc:ba:98:76:54, Dst: 12:34:56:78:9a:bc
Internet Protocol, Src Addr: 1.2.3.111 (1.2.3.111), Dst Addr: 1.2.3.210
(1.2.3.210)
Transmission Control Protocol, Src Port: sunrpc (111), Dst Port: 32781
(32781), Seq: 1, Ack: 61, Len: 32
    Source port: sunrpc (111)
    Destination port: 32781 (32781)
    Sequence number: 1 (relative sequence number)
    Next sequence number: 33 (relative sequence number)
    Acknowledgement number: 61 (relative ack number)
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
    Window size: 5792 (scaled)
    Checksum: 0x933b (incorrect, should be 0xca7b)
    Options: (12 bytes)
Remote Procedure Call, Type:Reply XID:0x295f366c
Portmap GETPORT Reply Port:2049 Port:2049

No. Time Source Destination Protocol
Info
     14 0.000660 1.2.3.210 1.2.3.111 NFS V3
NULL Call (Reply In 16)

Frame 14 (110 bytes on wire, 110 bytes captured)
Ethernet II, Src: 12:34:56:78:9a:bc, Dst: fe:dc:ba:98:76:54
Internet Protocol, Src Addr: 1.2.3.210 (1.2.3.210), Dst Addr: 1.2.3.111
(1.2.3.111)
Transmission Control Protocol, Src Port: 32782 (32782), Dst Port: 2049
(2049), Seq: 1, Ack: 1, Len: 44
    Source port: 32782 (32782)
    Destination port: 2049 (2049)
    Sequence number: 1 (relative sequence number)
    Next sequence number: 45 (relative sequence number)
    Acknowledgement number: 1 (relative ack number)
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
    Window size: 5840 (scaled)
    Checksum: 0x1c2c (correct)
    Options: (12 bytes)
Remote Procedure Call, Type:Call XID:0x453d111d
Network File System, NULL Call

No. Time Source Destination Protocol
Info
     16 0.000688 1.2.3.111 1.2.3.210 NFS V3
NULL Reply (Call In 14)

Frame 16 (94 bytes on wire, 94 bytes captured)
Ethernet II, Src: fe:dc:ba:98:76:54, Dst: 12:34:56:78:9a:bc
Internet Protocol, Src Addr: 1.2.3.111 (1.2.3.111), Dst Addr: 1.2.3.210
(1.2.3.210)
Transmission Control Protocol, Src Port: 2049 (2049), Dst Port: 32782
(32782), Seq: 1, Ack: 45, Len: 28
    Source port: 2049 (2049)
    Destination port: 32782 (32782)
    Sequence number: 1 (relative sequence number)
    Next sequence number: 29 (relative sequence number)
    Acknowledgement number: 45 (relative ack number)
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
    Window size: 5792 (scaled)
    Checksum: 0x9337 (incorrect, should be 0xa2d3)
    Options: (12 bytes)
Remote Procedure Call, Type:Reply XID:0x453d111d
Network File System, NULL Reply

No. Time Source Destination Protocol
Info
     24 0.000971 1.2.3.210 1.2.3.111 Portmap V2
GETPORT Call (Reply In 26) MOUNT(100005) V:3 TCP

Frame 24 (126 bytes on wire, 126 bytes captured)
Ethernet II, Src: 12:34:56:78:9a:bc, Dst: fe:dc:ba:98:76:54
Internet Protocol, Src Addr: 1.2.3.210 (1.2.3.210), Dst Addr: 1.2.3.111
(1.2.3.111)
Transmission Control Protocol, Src Port: 32783 (32783), Dst Port: sunrpc
(111), Seq: 1, Ack: 1, Len: 60
    Source port: 32783 (32783)
    Destination port: sunrpc (111)
    Sequence number: 1 (relative sequence number)
    Next sequence number: 61 (relative sequence number)
    Acknowledgement number: 1 (relative ack number)
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
    Window size: 5840 (scaled)
    Checksum: 0xc632 (correct)
    Options: (12 bytes)
Remote Procedure Call, Type:Call XID:0x5517656d
Portmap GETPORT Call MOUNT(100005) Version:3 TCP

No. Time Source Destination Protocol
Info
     26 0.001105 1.2.3.111 1.2.3.210 Portmap V2
GETPORT Reply (Call In 24) Port:1013

Frame 26 (98 bytes on wire, 98 bytes captured)
Ethernet II, Src: fe:dc:ba:98:76:54, Dst: 12:34:56:78:9a:bc
Internet Protocol, Src Addr: 1.2.3.111 (1.2.3.111), Dst Addr: 1.2.3.210
(1.2.3.210)
Transmission Control Protocol, Src Port: sunrpc (111), Dst Port: 32783
(32783), Seq: 1, Ack: 61, Len: 32
    Source port: sunrpc (111)
    Destination port: 32783 (32783)
    Sequence number: 1 (relative sequence number)
    Next sequence number: 33 (relative sequence number)
    Acknowledgement number: 61 (relative ack number)
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
    Window size: 5792 (scaled)
    Checksum: 0x933b (incorrect, should be 0xcf9c)
    Options: (12 bytes)
Remote Procedure Call, Type:Reply XID:0x5517656d
Portmap GETPORT Reply Port:1013 Port:1013

No. Time Source Destination Protocol
Info
     34 0.001390 1.2.3.210 1.2.3.111 MOUNT V3
NULL Call (Reply In 36)

Frame 34 (110 bytes on wire, 110 bytes captured)
Ethernet II, Src: 12:34:56:78:9a:bc, Dst: fe:dc:ba:98:76:54
Internet Protocol, Src Addr: 1.2.3.210 (1.2.3.210), Dst Addr: 1.2.3.111
(1.2.3.111)
Transmission Control Protocol, Src Port: 32784 (32784), Dst Port: 1013
(1013), Seq: 1, Ack: 1, Len: 44
    Source port: 32784 (32784)
    Destination port: 1013 (1013)
    Sequence number: 1 (relative sequence number)
    Next sequence number: 45 (relative sequence number)
    Acknowledgement number: 1 (relative ack number)
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
    Window size: 5840 (scaled)
    Checksum: 0xb835 (correct)
    Options: (12 bytes)
Remote Procedure Call, Type:Call XID:0x2e424981
Mount Service

No. Time Source Destination Protocol
Info
     36 0.001628 1.2.3.111 1.2.3.210 MOUNT V3
NULL Reply (Call In 34)

Frame 36 (94 bytes on wire, 94 bytes captured)
Ethernet II, Src: fe:dc:ba:98:76:54, Dst: 12:34:56:78:9a:bc
Internet Protocol, Src Addr: 1.2.3.111 (1.2.3.111), Dst Addr: 1.2.3.210
(1.2.3.210)
Transmission Control Protocol, Src Port: 1013 (1013), Dst Port: 32784
(32784), Seq: 1, Ack: 45, Len: 28
    Source port: 1013 (1013)
    Destination port: 32784 (32784)
    Sequence number: 1 (relative sequence number)
    Next sequence number: 29 (relative sequence number)
    Acknowledgement number: 45 (relative ack number)
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
    Window size: 5792 (scaled)
    Checksum: 0x9337 (incorrect, should be 0x3ee0)
    Options: (12 bytes)
Remote Procedure Call, Type:Reply XID:0x2e424981
Mount Service

No. Time Source Destination Protocol
Info
     44 0.001961 1.2.3.210 1.2.3.111 MOUNT V3
MNT Call (Reply In 46)

Frame 44 (198 bytes on wire, 198 bytes captured)
Ethernet II, Src: 12:34:56:78:9a:bc, Dst: fe:dc:ba:98:76:54
Internet Protocol, Src Addr: 1.2.3.210 (1.2.3.210), Dst Addr: 1.2.3.111
(1.2.3.111)
Transmission Control Protocol, Src Port: 905 (905), Dst Port: 1013 (1013),
Seq: 1, Ack: 1, Len: 132
    Source port: 905 (905)
    Destination port: 1013 (1013)
    Sequence number: 1 (relative sequence number)
    Next sequence number: 133 (relative sequence number)
    Acknowledgement number: 1 (relative ack number)
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
    Window size: 5840 (scaled)
    Checksum: 0xb4bb (correct)
    Options: (12 bytes)
Remote Procedure Call, Type:Call XID:0x5f175b37
Mount Service

No. Time Source Destination Protocol
Info
     46 0.007929 1.2.3.111 1.2.3.210 MOUNT V3
MNT Reply (Call In 44)

Frame 46 (126 bytes on wire, 126 bytes captured)
Ethernet II, Src: fe:dc:ba:98:76:54, Dst: 12:34:56:78:9a:bc
Internet Protocol, Src Addr: 1.2.3.111 (1.2.3.111), Dst Addr: 1.2.3.210
(1.2.3.210)
Transmission Control Protocol, Src Port: 1013 (1013), Dst Port: 905 (905),
Seq: 1, Ack: 133, Len: 60
    Source port: 1013 (1013)
    Destination port: 905 (905)
    Sequence number: 1 (relative sequence number)
    Next sequence number: 61 (relative sequence number)
    Acknowledgement number: 133 (relative ack number)
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
    Window size: 5792 (scaled)
    Checksum: 0x9357 (incorrect, should be 0x1744)
    Options: (12 bytes)
Remote Procedure Call, Type:Reply XID:0x5f175b37
Mount Service

No. Time Source Destination Protocol
Info
     57 3.161871 1.2.3.210 1.2.3.111 NFS V3
FSINFO Call, FH:0x03fa0008

Frame 57 (202 bytes on wire, 202 bytes captured)
Ethernet II, Src: 12:34:56:78:9a:bc, Dst: fe:dc:ba:98:76:54
Internet Protocol, Src Addr: 1.2.3.210 (1.2.3.210), Dst Addr: 1.2.3.111
(1.2.3.111)
Transmission Control Protocol, Src Port: 800 (800), Dst Port: 2049 (2049),
Seq: 1, Ack: 1, Len: 136
    Source port: 800 (800)
    Destination port: 2049 (2049)
    Sequence number: 1 (relative sequence number)
    Next sequence number: 137 (relative sequence number)
    Acknowledgement number: 1 (relative ack number)
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
    Window size: 5840 (scaled)
    Checksum: 0x81c4 (correct)
    Options: (12 bytes)
Remote Procedure Call, Type:Call XID:0x9a1b4739
Network File System, FSINFO Call DH:0x03fa0008

No. Time Source Destination Protocol
Info
     58 3.161879 1.2.3.111 1.2.3.210 TCP
2049 > 800 [ACK] Seq=1 Ack=137 Win=6864 Len=0 TSV=177229225 TSER=4294915050

Frame 58 (66 bytes on wire, 66 bytes captured)
Ethernet II, Src: fe:dc:ba:98:76:54, Dst: 12:34:56:78:9a:bc
Internet Protocol, Src Addr: 1.2.3.111 (1.2.3.111), Dst Addr: 1.2.3.210
(1.2.3.210)
Transmission Control Protocol, Src Port: 2049 (2049), Dst Port: 800 (800),
Seq: 1, Ack: 137, Len: 0
    Source port: 2049 (2049)
    Destination port: 800 (800)
    Sequence number: 1 (relative sequence number)
    Acknowledgement number: 137 (relative ack number)
    Header length: 32 bytes
    Flags: 0x0010 (ACK)
    Window size: 6864 (scaled)
    Checksum: 0x26fb (correct)
    Options: (12 bytes)
    SEQ/ACK analysis
        This is an ACK to the segment in frame: 57
        The RTT to ACK the segment was: 0.000008000 seconds

Relevant Pages

  • RE: Spoofed scans
    ... Sequence numbers would not be as ... target ADENA for an open TCP port 22. ... His plan is to get relative sequence numbers from PATSY while ... /* A couple of words on spoofing should be mentioned. ...
    (Incidents)
  • Re: Naming output files
    ... You will need to either set up a convoy orchestration to process one ... to see if there is a usable sequence number there. ... in the Message1 construct shape, then send Message1 out to a FILE send port: ...
    (microsoft.public.biztalk.general)
  • Re: block_ssh_guessers
    ... be bothered to set up certificates, google for port knocking. ... port-knocking--don't be bloody stupid. ... As I said in another message, a simple setup to change the sequence every ... than it is to launch a buffer overflow attack against a port that is just ...
    (comp.os.linux.security)
  • Re: Linux v2.6.22-rc3
    ... so the drive doesn't like the new SRST sequence. ... then the second one won't be given any timeout AT ... the first timeout seems to be for ANOTHER PORT ...
    (Linux-Kernel)