READ_DMA TIMEOUT/FAILURE's and g_vfs_done() I/O errors

Post Reply
jmasfitpc
Posts: 1
Joined: Wed Jun 10, 2009 5:34 pm

READ_DMA TIMEOUT/FAILURE's and g_vfs_done() I/O errors

Post by jmasfitpc »

Hi,

I manage two Fit-PC2's which are initial (pre) production.
Installed on one is FreeBSD 7.1-p5 and the other FreeBSD 7.2.
Both had the some problem, one after several weeks of up time
and the other after one week of up time. The kernel on both
started to log,

ad0: TIMEOUT - READ_DMA48 ... LBA=#

or,

ad0: TIMEOUT - READ_DMA ... LBA=#

and then g_vfs_done() ... error = 5 entries.

One machine could no long be accessed even from the
console and had to be hard reset. The other would eventually
let you on the console so it could be shutdown. An fsck was
done on both from single user and only the hard reset machine
had any issues, but they were normal for a hard reset. The
systems have been up now for a couple days now with no
problems.

Smartmontools is installed and the disk shows,

=== START OF INFORMATION SECTION ===
Device Model: ST9160310AS
Serial Number: 5SV040M2
Firmware Version: SD03
User Capacity: 160,041,885,696 bytes
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Wed Jun 10 13:00:12 2009 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 111 099 006 Pre-fail Always - 39299002
3 Spin_Up_Time 0x0003 099 099 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 29
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 100 253 030 Pre-fail Always - 172029
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 311
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 29
184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 12
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 063 055 045 Old_age Always - 37 (Lifetime Min/Max 35/39)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 61
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 1
193 Load_Cycle_Count 0x0032 053 053 000 Old_age Always - 95280
194 Temperature_Celsius 0x0022 037 045 000 Old_age Always - 37 (0 22 0 0)
195 Hardware_ECC_Recovered 0x001a 045 036 000 Old_age Always - 39299002
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0

Note that Raw_Read_Error_Rate, Seek_Error_Rate, Load_Cycle_Count,
and Hardware_ECC_Recovered are large, but Reallocated_Sector_Ct is
0. A short offline test shows no errors and it passes the overall-health
self-assessment test.

Any thoughts on why the sudden problems? Why after the
reboot and (clean) fsck the machines are now working fine?

Best Wishes

john

fluffycloud
Posts: 55
Joined: Tue Mar 24, 2009 5:32 pm

Re: READ_DMA TIMEOUT/FAILURE's and g_vfs_done() I/O errors

Post by fluffycloud »

They both failed at the same time? and the disk itself reports errors... recoverable, but actual errors? I'd look for an environmental problem if two seperate disks in seperate machines failed, temporarily, at the same point in time.

Your AC go out? Have an earthquake? etc...

-Fluffy

swelch@complex.org
Posts: 14
Joined: Wed Apr 22, 2009 4:59 pm

Re: READ_DMA TIMEOUT/FAILURE's and g_vfs_done() I/O errors

Post by swelch@complex.org »

Both machines were plugged into the same UPS (1500 Watt, 300 WH), and remember, they don't have the power-on reset feature, so there was no "real" power failure. Of course, that doesn't rule out a glitch getting through the power supply. There doesn't seem to be an event log on this UPS that I can find, but no evidence of a power event long enough to run this UPS down significantly with this small load was visible anywhere else in the server stack.

The machines were unattended at the time of the failure, and I don't know that we have determined if the failures were exactly simultaneous.

Gotta be a power or EMP event, I think, too. Machines still running fine with no sign of problems. Dunno if the soft errors are still building up, but that might be an interesting thing to watch.

Hmmm... earthquake? No visible damage, and looking at the USGS site, it doesn't look like there are any reported earthquakes in the Bay area Sunday afternoon / evening, when I think the machines went down. There were a few CA earthquakes in this time, but none that had reports of shaking anywhere within a hundred miles of these machines.

A mystery...

St

fluffycloud
Posts: 55
Joined: Tue Mar 24, 2009 5:32 pm

Re: READ_DMA TIMEOUT/FAILURE's and g_vfs_done() I/O errors

Post by fluffycloud »

Sorry, by AC i didn't mean "Alternating Current" but "Air Conditioning" ... i was refering to a heating problem.

The other environmental problem that could affect both is remote user load. If these machines were 'unattended', I assume you're using them for servers in some capacity, could a spike in user load have temporarily overwhelmed them, and put them into a 'bad state', or are they serving different user-bases?

-Fluffy

swelch@complex.org
Posts: 14
Joined: Wed Apr 22, 2009 4:59 pm

Re: READ_DMA TIMEOUT/FAILURE's and g_vfs_done() I/O errors

Post by swelch@complex.org »

Sorry, by AC i didn't mean "Alternating Current" but "Air Conditioning" ... i was refering to a heating problem.
Well, both units are on a PC laptop cooling pad, and only feel warm to the touch. Also, the log that John send earlier showed that the max temp of the disk drives was less than 40C, I think.
The other environmental problem that could affect both is remote user load. If these machines were 'unattended', I assume you're using them for servers in some capacity, could a spike in user load have temporarily overwhelmed them, and put them into a 'bad state', or are they serving different user-bases?
Well, one is a seldom used web server primarily, and the other is a pretty busy mail server...

It's puzzling, as Hal said...

St

Post Reply

Return to “FreeBSD”