ACPI Kernel issue

jan123
Posts: 7
Joined: Sat Dec 05, 2009 10:26 pm

Re: ACPI Kernel issue

Post by jan123 »

Hi,

I'm using Ubuntu 9.10 and I'm having the problem that the fit-pc2 freezes when operating with large amount of files. I was able to reproduce the problem when deleting about 15.000 small files via samba.

Symptoms on the windows 7 client (that deletes the files via samba):
- Delete dialog goes down to 0 elements per seconds and than an error like "file is not accessible" is shown
- Explorer shows error about lost samba connection (something like "server not available")
- Open ssh sessions with putty become inactive (something like "server ended connection unexpectedly")
- After some seconds and pressing F5 the explorer reconnects the share
- Ssh session can be opened again

Symptoms on the fit-pc2 server:
- Sometimes complete freeze (only hard resets helps)
- Sometimes the error "BUG: soft lockup - CPU#0 stuck for 61 sec!" is displayed again and again in the console

Btw: Similar problems occurred committing a large set of files into a svn (checkout and svn one the fit-pc) or cleaning a svn checkout. To me, it seems that this is a general problems working with large set of files and not a network specific issue.

Thanks to prj's post, setting "Processur full speed" to "GV3 Only" seems to solve / bypass the problems. I'm now able to delete >10.000 files without freezes.

At the moment, I'm using the default ubuntu kernel (2.6.31-16-generic) and not the fit-pc2-kernel. Are there any patches in the fit-pc2 kernel, that should fix the problem? My short tests with the fit-pc2 kernel results in the same error.

tkalmijn
Posts: 4
Joined: Mon Dec 07, 2009 8:21 am

Re: ACPI Kernel issue

Post by tkalmijn »

I have almost the exact same problem.

I run Ubuntu 9.10 server edition on fit pc2 (1.1ghz) with the 64gb SSD (Kingston).

My system freezes with file transfers from another Ubuntu (desktop) machine. I try to tranfer a bunch of files packaged as a tar using netcat. The tranfer is ok for about 9gb, after that the system freezes up and a reset of my fit pc2 is the only solution to make it "fit" again :-|

I used different protocols to see if that helps: smb, nc, scp, sftp.. they all give the same symptons, except that the nc command works fasted so it can tranfer more data before the system locks op.

I have tried a different io scheduler because I use the SSD by specifying 'elevator=noop' in grub.cnf. This should improve io for SSD's but it doesnt help my problem.

I thought it may have to do with the NIC. A RealTek r8168 driver is installed by Ubuntu automatically which is consistent with the actual NIC reported by "slpci". A new driver has been released by RealTek 17 dec, but I wasnt able to download and try it yet.

I will try the GV3 work around tonight and report back here. Although I doubt it will work for me as the processor load during the file transfer is only about 25% (as reported by 'top').

cheers,
Tom

morgad
Posts: 17
Joined: Thu Oct 15, 2009 8:13 pm

Re: ACPI Kernel issue

Post by morgad »

question - are you using gigabit ethernet?
we had similar problem at work doing big transfers on gigabit (with SuSe), switching to 100Mbit cured things

Dave

jan123
Posts: 7
Joined: Sat Dec 05, 2009 10:26 pm

Re: ACPI Kernel issue

Post by jan123 »

Hi,
yes I'm using gigabit ethernet and I did not try to limit the speed to 100Mbit.

Jan

Btw: I have another machine (not a fit-pc) freezing when transferring large amount of data. I will take your hint into account I'm going to examine the problem.

tkalmijn
Posts: 4
Joined: Mon Dec 07, 2009 8:21 am

Re: ACPI Kernel issue

Post by tkalmijn »

morgad

I am using Gigabit RealTek.

I switched to GV3 in the BIOS and my problems went away. I was able to netcat a 25gb file without problems.

Thanks to this forum I got my problem solved, I say: cheers!

Tom

prj
Posts: 90
Joined: Tue May 12, 2009 9:13 am

Re: ACPI Kernel issue

Post by prj »

I finally got around hooking a monitor to my FIT-PC2 to change some BIOS settings. I've been running in the GV3 state for quite some time and it works fine. Regardless it would be nice to figure out the real problem for the system hangs.

The c-states available on the CPU are: C0-C4 + C6. The C6 mode is called "Deep power down" and is a new technology from intel. It consumes about half the power (~80mW) of C4 (~160mW) [see Intel paper]. The C6 state isn't found by the kernel and seems to be what is causing the hangs. I've been running my kernel with "processor.max_cstate=4" for 24 hours and haven't experienced any hangs yet.

Limiting c-states to C4 seems to make the CPU run slightly cooler and consume less power (as mentioned above) than setting GV3 in the BIOS so it's a better fix. I'll digg some in the ACPI DSDT to see if the problem is there but I'm not very fluent in ASL so help is appreciated.

Cheers

pebenito
Posts: 6
Joined: Sun Nov 29, 2009 9:12 pm

Re: ACPI Kernel issue

Post by pebenito »

prj wrote:I finally got around hooking a monitor to my FIT-PC2 to change some BIOS settings. I've been running in the GV3 state for quite some time and it works fine. Regardless it would be nice to figure out the real problem for the system hangs.

The c-states available on the CPU are: C0-C4 + C6. The C6 mode is called "Deep power down" and is a new technology from intel. It consumes about half the power (~80mW) of C4 (~160mW) [see Intel paper]. The C6 state isn't found by the kernel and seems to be what is causing the hangs. I've been running my kernel with "processor.max_cstate=4" for 24 hours and haven't experienced any hangs yet.

Limiting c-states to C4 seems to make the CPU run slightly cooler and consume less power (as mentioned above) than setting GV3 in the BIOS so it's a better fix. I'll digg some in the ACPI DSDT to see if the problem is there but I'm not very fluent in ASL so help is appreciated.

Cheers
I tried this over the weekend, and it was fine until I tried to push a huge amount of data to the system last night over NFS, and then it hanged. It works fine with a max_cstate=2.

prj
Posts: 90
Joined: Tue May 12, 2009 9:13 am

Re: ACPI Kernel issue

Post by prj »

Bummer, I guess I haven't stressed my system enough. Two weeks of uptime without a single incident. Pebenito, can you test with max_cstate=3 since I'm having trouble hitting the bug? I did some more digging and powertop gives interesting output.

At max_cstate=4, powertop says:
Your CPU supports the following C-states : C1 C2 C4 C6
Your BIOS reports the following C-states : C1 C2 C4 C6

At max_cstate=3, powertop says:
Your CPU supports the following C-states : C1 C2 C4 C6
Your BIOS reports the following C-states : C1 C2 C4

This is in conflict of what is given in /sys/devices/system/cpu/cpu0/cpuidle/...

My system is in the lowest possible c-state 99.5% of the time so every c-state counts.

Cheers

pebenito
Posts: 6
Joined: Sun Nov 29, 2009 9:12 pm

Re: ACPI Kernel issue

Post by pebenito »

prj wrote:Bummer, I guess I haven't stressed my system enough. Two weeks of uptime without a single incident. Pebenito, can you test with max_cstate=3 since I'm having trouble hitting the bug? I did some more digging and powertop gives interesting output.

At max_cstate=4, powertop says:
Your CPU supports the following C-states : C1 C2 C4 C6
Your BIOS reports the following C-states : C1 C2 C4 C6

At max_cstate=3, powertop says:
Your CPU supports the following C-states : C1 C2 C4 C6
Your BIOS reports the following C-states : C1 C2 C4

This is in conflict of what is given in /sys/devices/system/cpu/cpu0/cpuidle/...
Why would setting a max of C3 make sense if the hardware doesn't support C3? You didn't say what you saw in /sys/..../cpuidle/ but presumably it only shows cstates up to max_cstate, whereas powertop queries the ACPI tables/hardware.

prj
Posts: 90
Joined: Tue May 12, 2009 9:13 am

Re: ACPI Kernel issue

Post by prj »

pebenito wrote: Why would setting a max of C3 make sense if the hardware doesn't support C3?
Good question. It's the terminology that is confusing. If we assume that powertop is correct, this is how it works:
First of all, C0 isn't considered a c-state when setting max_cstate. The integer parameter for max_cstate tell how many of the available c-states to allow. Setting max_cstate=2 makes C1 and C2 available. Setting max_cstate=3 makes C1, C2 and C4 available. Setting max_cstate=4 makes C1, C2, C4 and C6 available.
pebenito wrote:You didn't say what you saw in /sys/..../cpuidle/ but presumably it only shows cstates up to max_cstate, whereas powertop queries the ACPI tables/hardware.
It's even more confusing than that :)
If you look in the /sys/devices/system/cpu/cpu0/cpuidle directory:
At max_cstate=4, it will show 5 states: C0, C1, C2, C3, C4.
At max_cstate=3, it will show 4 states: C0, C1, C2, C3.
At max_cstate=2, it will show 3 states: C0, C1, C2

I would appreciate if you could test the max_cstate=3 since my setup don't allow me to push huge files at high speed.

Thanks

Post Reply

Return to “Linux on fit-PC2”