Filing downstream kernel bug reports

In order to automatically gather and attach necessary system information to a bug report, the preferred method for reporting a Ubuntu kernel bug is run the following command from a terminal window (Applications->Accessories->Terminal):

ubuntu-bug linux

The submitter should provide as much information as possible in the bug description:

The majority of kernel bug are hardware specific so be sure to note what hardware/device is being used.

Document any known steps to reproduce the bug.

Also note whether the bug exists in previous kernel versions of Ubuntu or if it's a regression from previous kernel versions.

If there are any questions about what information to provide, please contact a developer in the #ubuntu-kernel channel on the FreeNode IRC server for help on how to file a bug.

Adding debug info to a downstream bug report

If you didn't file your Ubuntu kernel bug using ubuntu-bug, please run the following at a terminal, which will automatically gather and attach the requested system information to the bug report:

apport-collect -p linux <bug#>

As a last resort, at a minimum, your bug report should contain the output of the following commands:

uname -a > uname-a.log

cat /proc/version_signature > version.log

dmesg > dmesg.log

sudo lspci -vvnn > lspci-vvnn.log

These four files should be attached separately to the bug report (not pasted into comments or tar and zipped). Please note that dmesg output should be captured as early as possible after bootup to avoid extraneous output.

Kernel sound bugs

If this is a sound related Ubuntu kernel bug, run the following to gather and attach important sound debugging information to the bug:

apport-collect -p alsa-base <bug#>

Problems in capturing information

Bootloader

If the bug occurs during bootup, you can disable the splash screen in one of the following ways:

Permanent: Edit /boot/grub/menu.lst and remove splash and quiet kernel parameters to bootloader from the entry for the buggy kernel

Temporary: Press 'Escape' key at the 3 second pause by Grub bootloader. Then press 'e' (edit) on the buggy kernel entry, followed by 'e' again on the kernel line. Then remove the splash and quiet keywords and press 'b' to boot.

Capturing OOPs

If the bug report involves a crash, it is hoped that a kernel backtrace (aka OOPS, kernel panic) is available. If the machine does not completely lockup from the crash, the backtrace should be available in the dmesg output. If the crash completely locks the system:

In X window mode

Sometimes crashes occur in X, and so terminal access is not available (to capture the kernel backtrace). When this occurs, the user should try to recreate the crash at the console (Ctrl+Alt+F1). If this is not possible, then annotate the bug as such.

Bug Triage

Triaging kernel bugs is a day-to-day effort, and can be very time consuming. Luckily, we have a lot of community members willing and able to help with this effort. The kernel team will also begin using a set of Kernel Arsenal scripts to help with the day-to-day triaging efforts. In order to make sure everyone working on kernel bugs follows the same policy, this document will describe how to handle the kernel bug workflow. This will also provide bug reporters with an idea of the life cycle their bug will follow.

Note that beginning with the Karmic development cycle an emphasis is being made to ensure bugs are tested and reported upstream. In relation to this, the Ubuntu kernel team will be transitioning their focus to fixing bugs which have been confirmed to exist upstream or are fixed upstream but exist in the Ubuntu kernel.

New Bugs

Bugs should always come in with a Status of New and an Importance of Undecided. Bugs should automatically have the appropriate debug information attached assuming they were reported using the preferred ubuntu-bug linux method. If this is the case, move the bug to a Confirmed state. If the bug is missing the appropriate debug information, the submitter should be asked to run apport-collect. Note the apport-collect command below only applies to Karmic.

apport-collect -p linux <bug#>

Set the bug to Incomplete. Also tag the bug "needs-kernel-logs". It's also a good idea to subscribe to a bug which you've set to Incomplete. That will ensure you are notified if and when the requested information is provided.

See process-new-bugs.py Kernel Arsenal script [to be linked soon]

Incomplete Bugs

Bugs are typically moved from a New state to an Incomplete state because they are lacking debug information necessary for the Ubuntu Kernel Team to debug the issue. Once all the information has been provided, the bug should be moved from an Incomplete state to a Confirmed state. If the bug was previously tagged "needs-kernel-logs", remove the tag as well once the bug moves to a Confirmed state.

If a bug is in an Incomplete state for more than 120 days and is not updated to provide the requested information, the bug should be expired by setting the status to Invalid and the reason it's being expired should be stated as a comment.

Confirmed Bugs

Confirmed bugs should have the appropriate debug information attached. In order for a Confirmed bug to move to a Triaged state, the upstream mainline kernel should be tested as well. This not only helps determine if the bug exists upstream, but also helps determine if a bug might be fixed upstream as well. If the bug exists upstream is also allow additional upstream developers to examine the issue. As mentioned previously, the Ubuntu kernel team will be focusing on bugs which have been confirmed to exist upstream or are fixed upstream but exist in the Ubuntu kernel. If a bug is in a Confirmed state but has not yet tested the upstream mainline kernel, tag the bug "needs-upstream-testing". If a bug has been tested with the upstream kernel, move the bug to a "Triaged" state. If the bug was previously tagged with "needs-upstream-testing", remove the tag once the bug moves to a Triaged state.

Triaged Bugs

Once a bug has been tested with the upstream mainline kernel and moved to a Triaged state, the bug should have their Importance set to something other than Undecided. If a bug is Triaged, then there should be enough information to know how important it is. If a bug was tested with the upstream mainline kernel and determined to exist upstream, a bug should also be reported upstream .

Note, you must be a member of the Ubuntu Bug Control team in order to set bugs to Triaged.

In Progress

A bug will move from Triaged to In Progress when a developer has chosen to actively work on the bug. If you are a developer and have set a bug to In Progress, make sure you also assign the bug to yourself. Please keep in mind that a bug marked In Progress for an extended period (greater than a week or two) should be updated periodically with any progress. If you are unable to dedicate time to working on the bug, move the bug back to a Triaged state and unassign yourself from the bug.

Also, do not assign another individual to a bug without their consent first! Doing so just gives bug reporters/subscribers a false sense that someone is actively working on a bug when that may not be the case. Leave it to the discretion of the developer to take ownership of a bug.

Fix Committed

Bugs that are marked Fix Committed are considered fixed by a patch which as been committed to the Ubuntu kernel git repository. This does not mean the fix has been released but it should be expected to be in the next kernel upload. There is no determinate time when a kernel upload will happen, it's up to the discretion of the Ubuntu Kernel Team.

Fix Released

Bugs are moved to a Fix Released state when the fix is readily available in the Ubuntu archive (in the updates pocket). If the Ubuntu kernel developer correctly created the git commit message to include the Launchpad Buglink, the launchpad janitor should automatically move a bug from Fix Committed to Fix Released once the fix is officially available.

Won't Fix

Won't Fix indicates the issue is recognized as a bug but a fix will not be applied, for ex BIOS issues. Some bugs may be marked as Won't Fix for a specific release and Fix Released against another.

Note, you must be a member of the Ubuntu Bug Control team in order to set bugs to Won't Fix.

Invalid

Bugs which are not legitimate bugs are set to Invalid. This can include bugs which failed to provide requested debug information or bugs which are the result of user error.

Tags

Tagging a bug is an easy way to group bugs across packages or within a single package. Tags which are commonly used for kernel bugs are:

Caveats

Sometimes kernel bugs are opened to track security vulnerabilities. These bugs usually contain the word "CVE" either in the title or bug description and will most likely have the ubuntu-security team subscribed to the bug. We should try to avoid spamming these bugs with comments to test the latest kernel to verify if the issue still exists. Whether manually posting to a bug or using python-launchpad-bugs to script comments, please take extra care to not cause more unnecessary traffic for the security team to deal with. Thanks.

Filing upstream kernel bugs

If your downstream bug report on Launchpad has been marked Triaged, and a Ubuntu community member asked you to read this, thank you for doing so! By reading and following this completely, you are maximizing the speed with which your bug will be fixed. Please take care to read every step carefully.

First step: Prepare the Kernel.org format information

Please ensure you follow the below format word for word. Just because you tested the latest mainline kernel, may have bisected a kernel regression, or others say they are experiencing the same problem, doesn't mean you should omit anything. Providing this information is vital for a developer to fix your problem, and to maximize the chance of your bug being addressed.

Please take care that when you provide the below information, check once again that you are booted into the newest available upstream mainline kernel only. Folks have had their bug marked Triaged, sent their e-mail, but had a newer kernel release come out without them realizing this. Hence, if upstream didn't ignore your report altogether, they will ask you to test this in the newest release that just came out and you didn't check for. Failure to do this will have negative unintended consequences. The Ubuntu kernel is the Ubuntu Community's responsibility, the upstream mainline kernel is the Kernel.org Community's responsibility, of which Ubuntu is a part of.

[1.] One line summary of the problem:
Please paste the Launchpad bug summary, and remove all references to Ubuntu and Ubuntu's kernels. The reason this is being reported upstream is because this is not just a downstream (Ubuntu) issue, but both an up and down one.

[2.] Full description of the problem/report: While booted into the newest mainline kernel only describe how the bug is reproducible in the latest mainline kernel only. If this is a regression, please note the specific commit.

[3.] Keywords (i.e., modules, networking, kernel): Please do not put anything here. This is how Kernel.org uses an undocumented keyword system, which has nothing to do with Ubuntu Tags system.

[4.] Kernel version (from /proc/version): While booted into the newest upstream mainline kernel only, please execute the following in a terminal and paste the results: cat /proc/version

[5.] Output of Oops.. message (if applicable) with symbolic information resolved (see Documentation/oops-tracing.txt)
This is only relevant if you had a oops crash (ex. flashing Caps Lock light). While booted into the newest upstream mainline kernel only, if you have a kernel oops, one may consult http://www.kernel.org/doc/Documentation/oops-tracing.txt . If this is too daunting please ask for help in your bug report, and of the Ubuntu community.

[6.] A small shell script or example program which triggers the problem (if possible)
This is for advanced community members. If you feel comfortable enough to do this, please do it. It is not crtical if this is beyond your skill level.

[7.] Environment
Please execute the following in a terminal, and paste the results: lsb_release -rd

[7.1.] Software (add the output of the ver_linux script here) While booted into the newest upstream mainline kernel only, this is found in the directory:
/usr/src/linux-headers-<VERSION>/scripts

where <VERSION> is the version of the kernel you are using, found in the directory /usr/src. You may run the script by changing to the directory via a terminal, and paste the results: sh ver_linux

[7.2.] Processor information (from /proc/cpuinfo): While booted into the newest upstream mainline kernel only, execute the following in a terminal, and paste the results: cat /proc/cpuinfo

[7.3.] Module information (from /proc/modules): While booted into the newest upstream mainline kernel only, execute the following in a terminal, and paste the results: cat /proc/modules

[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem) While booted into the newest upstream mainline kernel only, execute the following in a terminal, and paste the results: cat /proc/ioportscat /proc/iomem

[7.5.] PCI information ('lspci -vvv' as root) While booted into the newest upstream mainline kernel only, execute the following via a terminal, and paste the results: sudo lspci -vvv

[7.6.] SCSI information (from /proc/scsi/scsi) While booted into the newest upstream mainline kernel only, execute the following via a terminal, and paste the results: cat /proc/scsi/scsi

[7.7.] Other information that might be relevant to the problem (please look in /proc and include all information that you think to be relevant): While booted into the newest upstream mainline kernel only, execute the following via a terminal, and paste the results: ls /proc

As well, it is quite helpful that any further debugging information from http://www.kernel.org/doc/Documentation/ using the newest upstream mainline kernel available is provided.
In addition, please be 100% certain you have included all relevant comment information not included in the Bug Description and a kernel developer should review. They are some of the busiest people in the Linux space. They do not need to dive through downstream bug reports to find something that should have been provided when the bug was first filed. If you are unsure about anything asked for, or intend on not following this format, please do not file a report. Instead, continue to ask questions in Launchpad, and of the Ubuntu community, until all issues are cleared up.

Bluetooth problem only
Please execute the following in a terminal and post the results:
sudo cat /sys/kernel/debug/usb/devices

Second step: E-mail the maintainer mailinglist with Kernel.org format

Now that you have prepared the Kernel.org format information, the next step is identifying the proper upstream mailing list to e-mail your problem to.

Please note, when E-Mailing a maintainer mailing list, do so in plain text, not html. Otherwise, it will be flagged as SPAM, and silently discarded without notification to you. For more on this and other reasons your post would be flagged as SPAM, please see here.

Do not post any of the Kernel.org information as an attachment when e-mailing a maintainer mailing list. Instead, provide the information word for word in the body of your e-mail.

Reporting USB bugs upstream

Please ensure you post in your Launchpad report a URL of your e-mail from the mailing list found from http://vger.kernel.org/vger-lists.html. It can take a day or two for the e-mail to show up to the respective archive.

Reporting WiFi (802.11) USB dongle bugs upstream

If you have a wireless USB dongle bug, using the Kernel.org format, please:

Reporting non-WiFi USB dongle bugs upstream

First, please note that for non-wireless USB dongle bugs, the USB maintainer(s) do not want anyone to create a upstream bug report. For more on this, please see this upstream report comment. If you do open an upstream report anyways, it will be immediately closed, and cause negative unintended consequences. Instead, using the Kernel.org format, please:

Reporting non-USB bugs upstream

Not graphics card driver bug

If the bug is not in USB, the first step in reporting a bug upstream is to find the maintainer of the driver for the bug from the MAINTAINERS list. Here are some typical scenarios and suggested mailing list and procedures:

Suspend/resume, hibernate/resume, or brightness not working - linux-acpi (not linux-acpi-devel)

Touchpad or clickpad doesn't work at all - linux-input

Send an E-Mail to the mailing list indicated by the 'L:', and CC the maintainer (send an e-mail directly to the maintainer if no list is specified).

If this is a regression, CC the submitter of the regression commit.

Graphics card driver bug

Graphics card driver bugs are more traditional as upstream takes submissions into the respective bug tracker. Here is a breakdown of these:

Negative unintended consequences

If you contact upstream, please follow the kernel.org format created by kernel.org developers. Failure to follow these directions exactly as shown may have the following negative unintended consequences:

May likely result in your bug requiring upstream developers to ask unncessary follow up questions that should have been provided in the first place.

May likely result in your bug being promptly ignored by the kernel maintainer, submaintainer, or community developer(s) responsible to fix your kernel bug.

May create friction between you, the Ubuntu Community, and the kernel.org developers, which is not very Ubuntu.