Re: Filesystems acting as read-only

On 04/08/11 16:25, Aragorn wrote:
On Thursday 04 August 2011 16:06 in comp.os.linux.misc, David Brown
enlightened humanity with the following words...:

Can anyone think of a reason why a system would start reporting that
its file systems are read-only?

This is a remote system, and because it is involved in some traffic
routing (which is still working perfectly), I can't easily reboot it
at the moment, in case it does not restart smoothly.

Okay, remote system, so you don't always know what it's doing, and
presumably it has already been running the same kernel for a long time.


It's a Debian Etch system, with kernel 2.6.21.


Now, I'm not sure exactly when it changed, but there was a time that, in
the event of some filesystem damage on ext2/3 - e.d. due to an unclean
shutdown - the kernel would mount all disk-hosted filesystems read-only
to avoid corruption. I believe that this is no longer the case in
current kernels.

So my educated guess is that the machine was uncleanly shut down at some
point. Possibly a power failure or someone hard-reset the machine.


The uptime is 100 days, and this is a much more recent issue (there are log files from 31.07.2011).

There are several filesystems, all (except / and /boot) on logical disks. All are ext3, and all on the same disk.

So the problem seems to lie with the disk (or disk controller, I suppose).

Whenever I try to write to any file system, such as "touch /tmp/a", I
get an error "touch: cannot `a': Read-only file system". (The
exception is the tmpfs file system on /dev/shm, which works fine.)

Yep, because that's not a disk-hosted filesystem. It's RAM-hosted.


That's my conclusion too - the issue is at the disk level not the VFS level or anything related.

Commands (such as lsof) which involve writing temporary files are
failing with "Input/output error", which is perhaps not surprising if
they are also unable to write files.

"df" shows plenty of free space on all filesystems, "free" shows lots
of memory. "mount" shows the filesystems mounted read/write.

Hmm... "/bin/mount" reads from "/etc/mtab", and if the root filesystem
is mounted read-only, then the contents of "/etc/mtab" cannot be
updated.

A better indicator would be...

cat /proc/mounts


I tried that (based on Richard's post) - the filesystems have been re-mounted read-only. At least that matches the error messages!


One other thing that had happened was that there were thousands of
"nmbd" processes (part of samba). I cleared them up - I had to "kill
-9" them. I am wondering if there had been some runaway effect that
had led to these processes being spawned, and using up all of some
resource - despite having killed them afterwards.

This I cannot say, and presumably syslog will not be of much use either,
since it cannot be written to if the filesystems are mounted read-only.


That's my trouble!

.

Relevant Pages

  • Re: Filesystems acting as read-only
    ... presumably it has already been running the same kernel for a long time. ... caused allocation failures for the filesystems. ... I really hope the disk hasn't died - I have backups of the information, but it's still a big pain to re-install. ... A pragmatic approach might be to get a new disk, partition it up to an identical primary partition, dd the old data to it, fsck it..and try and boot from it. ...
    (comp.os.linux.misc)
  • Re: Filesystems acting as read-only
    ... its file systems are read-only? ... with kernel 2.6.21. ... There are several filesystems, all on logical ... So the problem seems to lie with the disk (or disk controller, ...
    (comp.os.linux.misc)
  • Re: Filesystems acting as read-only
    ... presumably it has already been running the same kernel for a long time. ... caused allocation failures for the filesystems. ... although the kernel log buffer is likely to have wrapped. ... I really hope the disk hasn't died - I have backups of the information, but it's still a big pain to re-install. ...
    (comp.os.linux.misc)
  • Re: How make partition more than 2TB?
    ... Does the kernel have support for larger than 2TB file systems? ... And exactly where did you get a disk with more than 2TB on it? ...
    (alt.os.linux)
  • OOPS: 2.6.23.11 in PadLock-AES when used in LRW-Mode
    ... while playing around with AES encrypted block devices on top of Software Raid-1/5 volumes I experienced a set of Kernel OOPSes which seem to be related to the current PadLock-AES driver. ... PadLock-AES in LRW mode also failed when layered directly above a raw disk partition. ... # Firmware Drivers ...
    (Linux-Kernel)