Hard disk problems / SMART help [LONG]

From: Captain Dondo (yan_at_NsOeSiPnAeMr.com)
Date: 11/21/05

Next message: Captain Dondo: "Re: Ecrix VXA-1 drive errors"
Previous message: juhani.jaakola_at_kolumbus.fi: "Re: Can't set IDE DMA with ASRock 939S56-M motherboard in kernel 2.6.14"
Next in thread: noi: "Re: Hard disk problems / SMART help [LONG]"
Reply: noi: "Re: Hard disk problems / SMART help [LONG]"
Reply: noi: "Re: Hard disk problems / SMART help [LONG]"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Date: Mon, 21 Nov 2005 05:44:21 -0800

I had a somewhat mysterious failure last night. I have a soft RAID array
that consists of 4 IDE drives on 2 controllers in master-slave pairs.
This array consists of 120 and 160 GB Seagate drives. The array is set up
as one large file system and exported via NFS.

Last night, the nfs server 'stopped responding'. A review of the log
files showed that hdc 'lost interrupt'. OK, fine. But smartctl shows
hda, not hdc, with errors.

And, to make things even more confusing, on reboot, both /dev/hdb and hdc
got 'timeout waiting for DMA' errors....

I find it hard to believe that I would lose 3 hard drives all at once. I
suspect that hda went down, and took down the other drives. Is this a
valid guess? Any other suggestions?

I don't really know how to interpret SMART results.... Any guidance there
would be appreciated....

***************************************************************

[root@tooth yan]# smartctl -a /dev/hda
bash: smartctl: command not found
[root@tooth yan]# /usr/sbin/smartctl -a /dev/hda
smartctl version 5.33 [i686-pc-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model: ST3160021A
Serial Number: 3JS0PDM3
Firmware Version: 3.06
User Capacity: 160,041,885,696 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 6
ATA Standard is: ATA/ATAPI-6 T13 1410D revision 2
Local Time is: Mon Nov 21 05:20:27 2005 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection: ( 430) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off supp ort.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
                                        No General Purpose Logging support.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 111) minutes.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_ FAILED RAW_VALUE
  1 Raw_Read_Error_Rate 0x000f 059 054 006 Pre-fail Always - 63368330
  3 Spin_Up_Time 0x0003 096 096 000 Pre-fail Always - 0
  4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 455
  5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
  7 Seek_Error_Rate 0x000f 081 060 030 Pre-fail Always - 126493885
  9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 15679
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 124
194 Temperature_Celsius 0x0022 028 059 000 Old_age Always - 28
195 Hardware_ECC_Recovered 0x001a 059 054 000 Old_age Always - 63368330
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 196 000 Old_age Always - 8
200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0

SMART Error Log Version: 1
ATA Error Count: 8 (device lER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 01 16 d6 da e0 Error: ICRC, ABRT 1 sectors at LBA = 0x00dad616 = 143416 54

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
  -- -- -- -- -- -- -- -- ---------------- --------------------
  25 00 80 97 d5 da e0 00 13:45:09.118 READ DMA EXT
  25 00 80 17 d5 da e0 00 13:45:09.117 READ DMA EXT
  25 00 80 97 d4 da e0 00 13:45:09.116 READ DMA EXT
  25 00 80 17 d4 da e0 00 13:45:09.114 READ DMA EXT
  25 00 80 97 d3 da e0 00 13:45:09.113 READ DMA EXT

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA _of_first_error
# 1 Extended offline Completed without error 00% 15672 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
    1 0 0 Not_testing
    2 0 0 Not_testing
    3 0 0 Not_testing
    4 0 0 Not_testing
    5 0 0 Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

yan@poseidon:~$ vi harddiskfail
yan@poseidon:~$ cat harddiskfail
[root@tooth yan]# smartctl -a /dev/hda
bash: smartctl: command not found
[root@tooth yan]# /usr/sbin/smartctl -a /dev/hda
smartctl version 5.33 [i686-pc-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART Error Log Version: 1
ATA Error Count: 8 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 8 occurred at disk power-on lifetime: 8207 hours (341 days + 23 hours)
When the command that caused the error occurred, the device was active or idle .

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 01 7e 0a cc e0 Error: ICRC, ABRT 1 sectors at LBA = 0x00cc0a7e = 133720 30

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
  -- -- -- -- -- -- -- -- ---------------- --------------------
  25 00 80 ff 09 cc e0 00 00:18:32.389 READ DMA EXT
  25 00 80 7f 09 cc e0 00 00:18:32.441 READ DMA EXT
  25 00 80 ff 08 cc e0 00 00:18:32.439 READ DMA EXT
  25 00 28 d7 08 cc e0 00 00:18:32.436 READ DMA EXT
  25 00 18 87 bc d2 e0 00 00:18:32.430 READ DMA EXT

Error 7 occurred at disk power-on lifetime: 8207 hours (341 days + 23 hours)
When the command that caused the error occurred, the device was active or idle .

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 01 fe 0d f8 e0 Error: ICRC, ABRT 1 sectors at LBA = 0x00f80dfe = 162565 10

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
  -- -- -- -- -- -- -- -- ---------------- --------------------
  25 00 38 c7 0d f8 e0 00 00:18:03.868 READ DMA EXT
  25 00 80 47 0d f8 e0 00 00:18:03.838 READ DMA EXT
  25 00 80 c7 0c f8 e0 00 00:18:03.833 READ DMA EXT
  25 00 18 27 30 ff e0 00 00:18:03.826 READ DMA EXT
  25 00 18 37 23 ff e0 00 00:18:03.821 READ DMA EXT

Error 6 occurred at disk power-on lifetime: 8207 hours (341 days + 23 hours)
When the command that caused the error occurred, the device was active or idle .

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 01 8e 01 48 e0 Error: ICRC, ABRT 1 sectors at LBA = 0x0048018e = 471899 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
  -- -- -- -- -- -- -- -- ---------------- --------------------
  25 00 18 77 01 48 e0 00 00:15:53.776 READ DMA EXT
  25 00 80 f7 00 48 e0 00 00:15:53.774 READ DMA EXT
  25 00 50 a7 00 48 e0 00 00:15:53.767 READ DMA EXT
  25 00 18 47 3b 65 e0 00 00:15:53.723 READ DMA EXT
  25 00 28 f7 20 65 e0 00 00:15:53.722 READ DMA EXT

Error 5 occurred at disk power-on lifetime: 8207 hours (341 days + 23 hours)
When the command that caused the error occurred, the device was active or idle .

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 01 f6 00 b4 e0 Error: ICRC, ABRT 1 sectors at LBA = 0x00b400f6 = 117967 26

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
  -- -- -- -- -- -- -- -- ---------------- --------------------
  25 00 18 df 00 b4 e0 00 00:13:27.057 READ DMA EXT
  25 00 38 a7 00 b4 e0 00 00:13:27.050 READ DMA EXT
  25 00 08 f7 0c ac e0 00 00:13:27.050 READ DMA EXT
  25 00 80 77 0c ac e0 00 00:13:27.048 READ DMA EXT
  25 00 80 f7 0b ac e0 00 00:13:27.046 READ DMA EXT

Error 4 occurred at disk power-on lifetime: 8205 hours (341 days + 21 hours)
When the command that caused the error occurred, the device was active or idle .

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 01 16 d6 da e0 Error: ICRC, ABRT 1 sectors at LBA = 0x00dad616 = 143416 54

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA _of_first_error
# 1 Extended offline Completed without error 00% 15672 -

LOG FILES

Nov 20 16:49:53 tooth kernel: hdc: status error: status=0x7f { DriveReady DeviceFault SeekComplete DataRequest CorrectedError Index Error }
Nov 20 16:49:53 tooth kernel: hdc: status error: error=0x7f { DriveStatusError UncorrectableError SectorIdNotFound TrackZeroNotFound AddrMarkNotFound }, LBAsect=260013951, sector=234436415
Nov 20 16:49:53 tooth kernel: ide: failed opcode was: unknown
Nov 20 16:49:53 tooth kernel: hdc: DMA disabled
Nov 20 16:49:53 tooth kernel: hdd: DMA disabled
Nov 20 16:49:53 tooth kernel: hdc: drive not ready for command
Nov 20 16:49:53 tooth kernel: ide1: reset: master: error (0x7f?)
Nov 20 16:49:53 tooth kernel: hdc: status error: status=0x7f { DriveReady DeviceFault SeekComplete DataRequest CorrectedError Index Error }
Nov 20 16:49:53 tooth kernel: hdc: status error: error=0x7f { DriveStatusError UncorrectableError SectorIdNotFound TrackZeroNotFound AddrMarkNotFound }, LBAsect=260013951, sector=234436415
Nov 20 16:49:53 tooth kernel: ide: failed opcode was: unknown
Nov 20 16:49:53 tooth kernel: hdc: drive not ready for command
Nov 20 16:49:57 tooth kernel: ide1: reset: success

................
At this point, the array went down - no response at all from it. cat /proc/mdstat would hang.

Nov 20 20:10:40 tooth kernel: hdc: lost interrupt
Nov 20 20:11:31 tooth kernel: hdc: lost interrupt
Nov 20 20:12:30 tooth kernel: hdc: lost interrupt
Nov 20 20:12:38 tooth sshd(pam_unix)[16281]: session opened for user yan by (uid=500)
Nov 20 20:12:55 tooth su(pam_unix)[16313]: session opened for user root by yan(uid=500)
Nov 20 20:13:30 tooth kernel: hdc: lost interrupt
Nov 20 20:14:30 tooth kernel: hdc: lost interrupt
Nov 20 20:16:30 tooth last message repeated 2 times
Nov 20 20:16:43 tooth sshd(pam_unix)[16437]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh
ruser= rhost=hermes.seiner.lan user=root
Nov 20 20:16:57 tooth sshd(pam_unix)[16446]: session opened for user yan by (uid=500)
Nov 20 20:17:30 tooth kernel: hdc: lost interrupt
Nov 20 20:18:15 tooth shutdown: shutting down for system reboot
Nov 20 20:18:30 tooth kernel: hdc: lost interrupt
Nov 20 20:19:30 tooth kernel: hdc: lost interrupt
Nov 20 20:21:30 tooth last message repeated 2 times
Nov 20 20:22:31 tooth last message repeated 2 times

----------------
on reboot, I try to set up the drives using hdparm

/sbin/hdparm -c 3 -m 16 -A 1 -a 32 -d 1 /dev/hda
/sbin/hdparm -c 3 -m 16 -A 1 -a 32 -d 1 /dev/hdb
/sbin/hdparm -c 3 -m 16 -A 1 -a 32 -d 1 /dev/hdc
/sbin/hdparm -c 3 -m 16 -A 1 -a 32 -d 1 /dev/hdd

This resulted in the following errors:

Nov 20 20:29:20 tooth kernel: hdb: dma_timer_expiry: dma status == 0x60
Nov 20 20:29:20 tooth kernel: hdb: DMA timeout retry
Nov 20 20:29:20 tooth kernel: hdb: timeout waiting for DMA
Nov 20 20:29:40 tooth kernel: hdb: dma_timer_expiry: dma status == 0x60
Nov 20 20:29:40 tooth kernel: hdb: DMA timeout retry
Nov 20 20:29:40 tooth kernel: hdb: timeout waiting for DMA
Nov 20 20:30:01 tooth kernel: hdc: dma_timer_expiry: dma status == 0x60
Nov 20 20:30:01 tooth kernel: hdc: DMA timeout retry
Nov 20 20:30:01 tooth kernel: hdc: timeout waiting for DMA
Nov 20 20:30:21 tooth kernel: hdc: dma_timer_expiry: dma status == 0x60
Nov 20 20:30:21 tooth kernel: hdc: DMA timeout retry
Nov 20 20:30:21 tooth kernel: hdc: timeout waiting for DMA

Next message: Captain Dondo: "Re: Ecrix VXA-1 drive errors"
Previous message: juhani.jaakola_at_kolumbus.fi: "Re: Can't set IDE DMA with ASRock 939S56-M motherboard in kernel 2.6.14"
Next in thread: noi: "Re: Hard disk problems / SMART help [LONG]"
Reply: noi: "Re: Hard disk problems / SMART help [LONG]"
Reply: noi: "Re: Hard disk problems / SMART help [LONG]"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Relevant Pages

Re: init using high cpu
... > hdc: ... drive not ready for command ... ESPECIALLY if any of the programs that init calls are stored on those ... buggy drives. ...
(comp.os.linux.setup)
Re: Trouble using some (fast) compact flash as ide device on an embedded system
... What IDE hardware / host driver is used by this system? ... hdc: status error: status=0x58 {DriveReady SeekComplete DataRequest} ... drive not ready for command ...
(Linux-Kernel)
Re: [PATCH] block: fix residual byte count handling
... hdc: status error: status=0x58 ... ide: failed opcode was: unknown ... drive not ready for command ...
(Linux-Kernel)
Re: cant mount drive
... So after running the command as root this ... >> capacity driver identify media model settings ... So either you do a manual 'cat ... [root@obsidian hdc]# cat capacity ...
(Fedora)