Tuesday, April 6, 2010

A soft lockup is the symptom of a task or kernel thread using and not releasing a CPU for a period of time (the softlockup_thresh setting ). User space processes should not be able to soft lockup a CPU.

How is a soft lockup detected ?

The Linux kernel creates a watchdog process for each CPU in the system. This should be visible using in the standard "ps" command, this is shown as [watchdog/N], where N is the number of the logical CPU.

This watchdog process/thread wakes up once a second, gets the current time stamp for the CPU it is running on and saves it into a per-CPU structure.

The softlockup_tick() function that gets called from the timer interrupt(). This function gets the current time stamp for the specified CPU and compares it to the saved one in the per-CPU structure.

If the current time stamp is more than softlockup_thresh seconds later than the saved time stamp then it's because the watchdog thread has not been run recently and a soft lockup message is generated on the console. As the timer counter on each CPU can be slightly slow or fast the counter is compared to the previous tick.

Why was the process/task allowed to hog the CPU ?

Not all code that runs on the CPU is considered to be a process. Tasklets / Interrupt handlers and blah are kernel functions that do not show up in the standard process listing (with the ps command).

The kernel still allocates them CPU time, but trusts them to relinquish control of the CPU.

Common causes of not relinquishing control.

Software bugs can cause the process/code to not relinquish the control of the CPU. The code could be waiting on a lock, or may be running code which continues in an infinite loop.

The other problem may be that the scheduler has ignored the process and the process has not been removed from the CPU for some timeframe. There are a number of kernel options that can isolate a cpu and run a single process on it. As the process "hogs" the CPU it may be seen as causing a soft lockup.

Note the lack of values beneath "Call Trace:" above. If a userspace process is the process that has not been scheduled/removed from the cpu no backtrace will be shown, only the registers and the process name.

If a kernel thread is responsible the backtrace for the kernel thread will be shown in the soft lockup message, however the backtrace includes code that shows the stack message. The DWARF2 undwinder attempts to show the actual stack trace. (shown above in bold).

Sometimes the call trace may only show the symptom of the soft lockup, and other kernel tasks will need to be investigated to show the cause.

Thursday, April 1, 2010

In some circumstances when attempting to debug a vmcore from Linux, you may have only been able to get part of the vmcore, either due a technical issue or the machine being forcebly rebooted before completion.

When loading the vmcore in crash you may find something similar to the message below: