So you've decided to do kernel bug triage. AWESOME! We could use the help. If you're already familiar with bugzilla and the kernel, here is a quickstart set of steps to help us triage the existing bugs. If you aren't familiar with kernel bugs, or need a refresher, the rest of the page has some good background on various aspects of kernel triage and we encourage you to read it before diving in.

+

+

Steps (Start at the top, work to the bottom):

+

+

* Read over the bug and come up with a basic classification (kernel panic, boot issue, RFE, missing/broken functionality, etc). Add a comment to the bug that provides this classification.

** If it is a bug in the video subsystem, reassign it to xorg-x11-drv-{ati,intel,nouveau}. Although the modesetting drivers live in the kernel, they are maintained by the same people who maintain our userspace graphics drivers. See [[KernelBugTriage#Video_Subsystem_bugs]]

+

** If it is in one of these categories, please add the matching email address as a "CC" on the bug:

* For oopses, add the oopsing function to the end of the bug subject. E.g. IP:hci_send_sco. For things like __list_del_entry, which are common kernel functions, try to find the latest function that was called before that and use that as the suffix.

+

* Look for duplicate bugs across all Fedora releases. If you find a duplicate, use whichever bug has the most information as the "active" bug and duplicate the other in bugzilla. If you duplicate a bug from a newer release against one in an older release, then change the version on the active bug to the newest version the issue has been seen on.

+

* If the bug is new, try to determine which kernel version it was first seen with. Note this in the bugzilla whiteboard with "first=X.Y.Z" (e.g. first=3.1.0). Also try to determine which kernel version is the latest tested and still having the issue. Note this in the bugzilla whiteboard with "latest=X.Y.Z" (e.g. latest=3.8.7).

+

* If the bug is missing information, ask for it and put the bug in NEEDINFO. This can include things like dmesg, modules in use, full panic/oops backtraces, or questions as to what was occurring on the machine at the time. Information should be attached as text/plain attachments, or simple comments in the bug if the information is fairly short. In order to avoid information overload, please ask for only relevant missing information (e.g. we don't need /proc/cpuinfo for wifi bugs, etc).

+

* If the bug looks fairly straightforward, try and see if upstream knows about it.

+

** Look on the various mailing lists and kernel.org bugzilla to see if something similar has been reported upstream (or google search). If so, add a reference to the Fedora bug in the thread or in the kernel.org bug.

+

** If it doesn't appear to be reported upstream, use the MAINTAINERS and scripts/get_maintainers.pl files in the kernel tree to narrow down who to report the issue to. Send them an email with all the relevant details from the bug, with kernel-team@fedoraproject.org on CC. Add a link to the mailing list thread archive of the email you sent to the Fedora bug, or use the external references field to pair it with a kernel.org bug.

+

* Mark the bug as Triaged with the bugzilla Triaged keyword.

+

+

== Kernel Bug Classification ==

+

[[KernelBugClassification]]

+

+

== Kernel Bugzilla Basics ==

+

+

* If there's an upstream (http://bugzilla.kernel.org) bug that matches

+

** In the Fedora bug, set 'external bugzilla references' to point to the upstream bug number

+

** In the upstream kernel.org bugzilla, add a comment along the lines of "also seen in Fedora. http://bugzilla.redhat.com/show_bug.cgi?id=253096"

+

* If the bug is against an already released Fedora (Ie, Fedora 7) and there isn't much information to go on "my machine locked up" for eg, request that the user try to reproduce the problem using the kernel-debug kernel. The additional debugging checks may trigger some clues.

+

* If the bug you are triaging contains a patch, please add [PATCH] to the beginning of the summary line of the bugzilla.

+

* Sometimes a user will attach a kernel panic as a jpeg, or in worst case, as a tarball of their /var/log/messages.

+

** It saves the kernel team time if the kernel oops parts of these are transcribed/cut out and pasted into the bug as text.

+

* Additionally, making sure that text attachments of bugs have their MIME type set to text/plain can save some time.

+

* You can ask reporters to try and duplicate with the latest nightly live composes: http://alt.fedoraproject.org/pub/alt/nightly-composes/

+

* You can ask reporters to try the newest kernel available in koji: http://koji.fedoraproject.org/koji/packageinfo?packageID=8

+

* If the bug appears to be video/kernel modesetting related, reassign it to the xorg-x11-drv-{intel,ati,nouveau} component. See [[KernelBugTriage#Video_Subsystem_bugs]]

+

* If the reporter is using a custom compiled kernel, close the bug as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=126342

+

+

== Bug state transitions ==

+

+

* A bug marked as MODIFIED has patches in testing and should not have their status changed. An exception to this is when a bug has been in MODIFIED state for some time (this usually indicates that the issue was fixed in an update, and no-one ever closed the bug. If in doubt, ping the reporter to retest with the latest build).

+

* If a bug has been in NEEDINFO for several weeks with no follow-up from the reporter, there's not a great deal we can do. In this situation, closing the bug is the only recourse available (which does actually tend to 'wake up' some reporters who then reopen the bug). See [[KernelBugTriage#Stale_Bugs]] below.

+

* When marking bugs as duplicates, we want the one with the most information to be the one that remains open, even if this means a higher numbered bug gets closed. Also, the Fedora release branches are kept on essentially the same kernel version. Therefore, if the same bug is reported in both releases then we close one of them as a duplicate of the other following the aforementioned rule. Duplicating bugs across releases is one way to control the bug count.

+

* Don't close bugs marked beginning with [mmconf] or [msi] (bugs beginning with [$something] are usually specifically marked so developers can quickly see the main issue within the bug)

+

+

== NEW state bugs ==

+

+

If the bug is new and (this type of bug must be key-worded in either WHITEBOARD or BLOCKER):

+

Hello,

+

+

:''Thank you for taking the time to report this bug. I have added the proper keyword to this bug to bring it to the attention of the <subsystem-name> subsystem maintainer.

:''Thank you for taking the time to report this bug. Bugs involving the installation process cannot usually be resolved after release date. We therefore ask reporters to use one (or both) of the following options:

:''Thank you for taking the time to report this bug. Bugs involving the installation process cannot usually be resolved after release date. We therefore ask reporters to use one (or both) of the following options:

−

:''1. Test with a Fedora Unity re-spin. These can be found at http://spins.fedoraunity.org/spins and include updates for installer bugs which may resolve the problem for you.''

+

:''Test with an Alpha, Beta or Release Candidate for the next version. These can be found at http://fedoraproject.org/get-prerelease and by testing these we can work to resolve any installation issues so that the final release is free of such bugs.''

−

:''2. Test with an Alpha, Beta or Release Candidate for the next version. These can be found at http://fedoraproject.org/get-prerelease and by testing these we can work to resolve any installation issues so that the final release is free of such bugs.''

+

:''However if the problem no longer exists then please close this bug or it will be closed in a few weeks if there is no additional information lodged.''

:''However if the problem no longer exists then please close this bug or it will be closed in a few weeks if there is no additional information lodged.''

−

{{admon/important|Outdated fedoraunity spins|The spins at fedoraunity are not updated since 2010-03-03.}}

{{admon/important|get-prerelease link|The ''get-prerelease'' link sometimes is not available or have not updated development releases.}}

{{admon/important|get-prerelease link|The ''get-prerelease'' link sometimes is not available or have not updated development releases.}}

Line 36:

Line 197:

:''This bug is in a video subsystem that has a kernel part. We track and work on these bugs via the driver package name instead of leaving them assigned to the kernel''

:''This bug is in a video subsystem that has a kernel part. We track and work on these bugs via the driver package name instead of leaving them assigned to the kernel''

−

== NEW state bugs ==

+

== Rebase Regressions ==

+

The kernel is often rebased to the latest stable kernel release and pushed as an update. This fixes a large amount of bugs, but it inevitably introduces a few regressions. If a bug is reported and it is determined that it worked with the previous kernel release, then a rebase regression entry should be added to the whiteboard field in the bug. This entry takes the form of:

−

If the bug is new and (this type of bugs must be key-worded in either WHITEBOARD or BLOCKER):

+

<pre>rebase-regression-<kernel version></pre>

−

Hello,

+

−

:''Thank you for taking the time to report this bug. I have added the proper keyword to this bug to bring it to the attention of the <subsystem-name> subsystem maintainer.

+

E.g. if there is a regression when moving to the 3.1 kernel, it should be:

−

:''Can you attach the following information to this bug: ''

+

<pre>rebase-regression-3.1</pre>

−

Then pick the info that is useful in this case:

+

−

* smolt profile

+

The latest working Fedora kernel version and the earliest broken version should be noted in the comment section if possible.

−

* dmesg output from boot before issue.

+

−

* dmesg output after issue.

+

−

* dmesg output from boot with 'quiet' option replaced with 'debug'

+

−

* lspci output

+

−

* lsusb output

+

−

For other necessary information check:

+

This will allow the kernel developers to track regressions introduced by a rebased kernel. That will also help facilitate our communication with the upstream kernel developers, as we can report these bugs to them to be included in the regression lists that are published upstream.

:''However, you can report it to the vendor of your non-free module.''

:''However, you can report it to the vendor of your non-free module.''

−

== Bug state transitions ==

+

== Stale Bugs ==

−

* A bug marked as MODIFIED has patches in testing and should not have their status changed. An exception to this is when a bug has been in MODIFIED state for some time (this usually indicates that the issue was fixed in an update, and no-one ever closed the bug. If in doubt, ping the reporter to retest with the latest build).

+

If a bug has been in NEEDINFO with no response from the reporter for 2 weeks, close the bug out with INSUFFICIENT_DATA and the following text:

−

* If a bug has been in NEEDINFO for several months with no follow-up from the reporter, there's not a great deal we can do. In this situation, closing the bug is the only recourse available (which does actually tend to 'wake up' some reporters who then reopen the bug).

+

−

* When marking bugs as duplicates, we want the one with the most information to be the one that remains open, even if this means a higher numbered bug gets closed.

+

−

* Don't close bugs marked beginning with [mmconf] or [msi] (bugs beginning with [$something] are usually specifically marked so developers can quickly see the main issue within the bug)

+

−

== Basics ==

+

:''This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 2 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.''

−

* If there's an upstream (http://bugzilla.kernel.org) bug that matches

+

{{Anchor|common_problems}}

−

* In the Fedora bug, set 'external bugzilla references' to point to the upstream bug number

+

−

* In the upstream kernel.org bugzilla, add a comment along the lines of "also seen in Fedora. http://bugzilla.redhat.com/show_bug.cgi?id=253096"

+

−

* If the report contains a lockdep trace, mark it as blocking FCMETA_LOCKDEP

+

−

* If the bug is against an already released Fedora (Ie, Fedora 7) and there isn't much information to go on "my machine locked up" for eg, request that the user try to reproduce the problem using the kernel-debug kernel. The additional debugging checks may trigger some clues.

+

−

* If the bug you are triaging contains a patch, please add [PATCH] to the beginning of the summary line of the bugzilla.

+

−

* Sometimes a user will attach a kernel panic as a jpeg, or in worst case, as a tarball of their /var/log/messages.

+

−

* It saves the kernel team time if the kernel oops parts of these are transcribed/cut out and pasted into the bug as text.

+

−

* Additionally, making sure that text attachments of bugs have their MIME type set to text/plain can save some time.

+

−

* You can ask reporters to try and duplicate with the latest nightly live composes: http://alt.fedoraproject.org/pub/alt/nightly-composes/

+

−

* You can ask reporters to try the newest kernel available in koji: http://koji.fedoraproject.org/koji/packageinfo?packageID=8

+

−

* If the bug appears to be video/kernel modesetting related, reassign it to the xorg-x11-drv-{intel,ati,nouveau} component.

+

−

* If the reporter is using a custom compiled kernel, close the bug as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=126342

+

−

{{Anchor|common_problems}}

== Common Problems ==

== Common Problems ==

Point reporters to: [[KernelCommonProblems]]

Point reporters to: [[KernelCommonProblems]]

−

−

== Bug assignment ==

−

For certain categories of bug, assigning/cc'ing people responsible for the relevant subsystem is a good idea.

Quick links to bug lists

Triage Quickstart

So you've decided to do kernel bug triage. AWESOME! We could use the help. If you're already familiar with bugzilla and the kernel, here is a quickstart set of steps to help us triage the existing bugs. If you aren't familiar with kernel bugs, or need a refresher, the rest of the page has some good background on various aspects of kernel triage and we encourage you to read it before diving in.

Steps (Start at the top, work to the bottom):

Read over the bug and come up with a basic classification (kernel panic, boot issue, RFE, missing/broken functionality, etc). Add a comment to the bug that provides this classification.

Determine which kernel subsystem or driver is involved. Prefix the bug subject with the driver/subsytem (e.g. ALSA, USB, ACPI, etc).

If it is a bug in the video subsystem, reassign it to xorg-x11-drv-{ati,intel,nouveau}. Although the modesetting drivers live in the kernel, they are maintained by the same people who maintain our userspace graphics drivers. See KernelBugTriage#Video_Subsystem_bugs

If it is in one of these categories, please add the matching email address as a "CC" on the bug:

Subsystem/Driver

CC

ACPI

fedora-kernel-acpi@fedoraproject.org

Audit

fedora-kernel-audit@fedoraproject.org

block (Issues with block/* OR drivers/block/loop.c)

fedora-kernel-block@fedoraproject.org

DMAR (Intel IOMMU)

fedora-kernel-dmar@fedoraproject.org

Ethernet (all issues)

fedora-kernel-ethernet@fedoraproject.org

Ethernet (bnx2, bnx2x, b44, tg3)

fedora-kernel-ethernet-broadcom@fedoraproject.org

Ethernet (r8169)

fedora-kernel-ethernet-realtek@fedoraproject.org

Filesystems (AIO, fs/aio.c)

fedora-kernel-aio@fedoraproject.org

Filesystems (BTRFS)

fedora-kernel-btrfs@fedoraproject.org

Filesystems (CIFS)

nfs-maint@redhat.com

Filesystems (DirectIO, fs/direct-io.c)

fedora-kernel-directio@fedoraproject.org

Filesystems (EXT2/EXT3/EXT4/JBD)

fedora-kernel-extfs@fedoraproject.org

Filesystems (fs/buffer.c)

fedora-kernel-fsbuffer@fedoraproject.org

Filesystems (NFS)

nfs-maint@redhat.com

Filesystems (XFS)

fedora-kernel-xfs@fedoraproject.org

Firewire

fedora-kernel-firewire@fedoraproject.org

Graphics (DRM core, but not Radeon/Intel/Nouveau drivers)

fedora-kernel-drm@fedoraproject.org

HID/Input/Multitouch (but not WACOM)

fedora-kernel-input@fedoraproject.org

intel-pstate (cpufreq driver)

fedora-kernel-intelpstate@fedoraproject.org

KVM virtualization

fedora-kernel-kvm@fedoraproject.org

LVM and device-mapper

lvm-team@redhat.com

Networking (NFC)

fedora-kernel-nfc@fedoraproject.org

Networking (TCP, UDP, SCTP, IP)

fedora-kernel-networking@fedoraproject.org

OpenVSwitch

fedora-kernel-openvswitch@fedoraproject.org

ptrace and utrace

fedora-kernel-ptrace@fedoraproject.org

PCI / PnP resource allocation

fedora-kernel-pci@fedoraproject.org

RAID (drivers/md)

fedora-kernel-raid@fedoraproject.org

SATA and libata

fedora-kernel-ata@fedoraproject.org

SCSI and libsas

fedora-kernel-scsi@fedoraproject.org

SELinux

fedora-kernel-selinux@fedoraproject.org

UEFI

fedora-kernel-uefi@fedoraproject.org

USB Video Cameras

fedora-kernel-usb-cameras@fedoraproject.org

Video4Linux

fedora-kernel-v4l@fedoraproject.org

Xen virtualization

fedora-kernel-xen@fedoraproject.org

Wireless (generic issues, or issues that do not match one of the below drivers)

fedora-kernel-wireless@fedoraproject.org

Wireless (ath*k)

fedora-kernel-wireless-ath@fedoraproject.org

Wireless (Broadcom (b43))

fedora-kernel-wireless-b43@fedoraproject.org

Wireless (brcm80211)

fedora-kernel-wireless-brcm80211@fedoraproject.org

Wireless (Intel (iwlwifi and iwlegacy (iwl3945 and iwl4965)))

fedora-kernel-wireless-iwl@fedoraproject.org

Wireless (Ralink (rt2x00))

fedora-kernel-wireless-ralink@fedoraproject.org

Wireless (Realtek (rtlwifi, r8712u))

fedora-kernel-wireless-realtek@fedoraproject.org

If multiple categories apply, add them each to the CC list.

For oopses, add the oopsing function to the end of the bug subject. E.g. IP:hci_send_sco. For things like __list_del_entry, which are common kernel functions, try to find the latest function that was called before that and use that as the suffix.

Look for duplicate bugs across all Fedora releases. If you find a duplicate, use whichever bug has the most information as the "active" bug and duplicate the other in bugzilla. If you duplicate a bug from a newer release against one in an older release, then change the version on the active bug to the newest version the issue has been seen on.

If the bug is new, try to determine which kernel version it was first seen with. Note this in the bugzilla whiteboard with "first=X.Y.Z" (e.g. first=3.1.0). Also try to determine which kernel version is the latest tested and still having the issue. Note this in the bugzilla whiteboard with "latest=X.Y.Z" (e.g. latest=3.8.7).

If the bug is missing information, ask for it and put the bug in NEEDINFO. This can include things like dmesg, modules in use, full panic/oops backtraces, or questions as to what was occurring on the machine at the time. Information should be attached as text/plain attachments, or simple comments in the bug if the information is fairly short. In order to avoid information overload, please ask for only relevant missing information (e.g. we don't need /proc/cpuinfo for wifi bugs, etc).

If the bug looks fairly straightforward, try and see if upstream knows about it.

Look on the various mailing lists and kernel.org bugzilla to see if something similar has been reported upstream (or google search). If so, add a reference to the Fedora bug in the thread or in the kernel.org bug.

If it doesn't appear to be reported upstream, use the MAINTAINERS and scripts/get_maintainers.pl files in the kernel tree to narrow down who to report the issue to. Send them an email with all the relevant details from the bug, with kernel-team@fedoraproject.org on CC. Add a link to the mailing list thread archive of the email you sent to the Fedora bug, or use the external references field to pair it with a kernel.org bug.

If the bug is against an already released Fedora (Ie, Fedora 7) and there isn't much information to go on "my machine locked up" for eg, request that the user try to reproduce the problem using the kernel-debug kernel. The additional debugging checks may trigger some clues.

If the bug you are triaging contains a patch, please add [PATCH] to the beginning of the summary line of the bugzilla.

Sometimes a user will attach a kernel panic as a jpeg, or in worst case, as a tarball of their /var/log/messages.

It saves the kernel team time if the kernel oops parts of these are transcribed/cut out and pasted into the bug as text.

Additionally, making sure that text attachments of bugs have their MIME type set to text/plain can save some time.

Bug state transitions

A bug marked as MODIFIED has patches in testing and should not have their status changed. An exception to this is when a bug has been in MODIFIED state for some time (this usually indicates that the issue was fixed in an update, and no-one ever closed the bug. If in doubt, ping the reporter to retest with the latest build).

If a bug has been in NEEDINFO for several weeks with no follow-up from the reporter, there's not a great deal we can do. In this situation, closing the bug is the only recourse available (which does actually tend to 'wake up' some reporters who then reopen the bug). See KernelBugTriage#Stale_Bugs below.

When marking bugs as duplicates, we want the one with the most information to be the one that remains open, even if this means a higher numbered bug gets closed. Also, the Fedora release branches are kept on essentially the same kernel version. Therefore, if the same bug is reported in both releases then we close one of them as a duplicate of the other following the aforementioned rule. Duplicating bugs across releases is one way to control the bug count.

Don't close bugs marked beginning with [mmconf] or [msi] (bugs beginning with [$something] are usually specifically marked so developers can quickly see the main issue within the bug)

NEW state bugs

If the bug is new and (this type of bug must be key-worded in either WHITEBOARD or BLOCKER):
Hello,

Thank you for taking the time to report this bug. I have added the proper keyword to this bug to bring it to the attention of the <subsystem-name> subsystem maintainer.

Bug assignment

Assigning/cc'ing people responsible for the relevant subsystem is a good idea.

When reassigning bugs to yourself, add kernel-maint@redhat.com to Cc:

If you are an upstream developer, and would like to be added to the Cc: of relevant bugs, send an email to kernel-team at fedoraproject.org,
(We used to maintain this here on the wiki, but for a number of reasons, we now maintain this list privately)

If the bug is against a previous point release kernel, and the reporter hasn't re-tested (this type of bugs must be set in NEEDINFO_REPORTER state):

Thank you for taking the time to report this bug. Updates to the kernel have been released since it was first reported. If you have time to update the package and re-test, please do so and report the results here. You can obtain the updated package by typing 'yum update kernel' or using the graphical updater, Software Update. If the problem no longer exists then please close this bug or it will be closed in a few weeks if there is no additional information lodged.

Installer bugs

If the bug refers to problems when installing Fedora in any of the possible ways CD, DVD, USB, network, etc (this type of bugs must be set in NEEDINFO_REPORTER state):

Thank you for taking the time to report this bug. Bugs involving the installation process cannot usually be resolved after release date. We therefore ask reporters to use one (or both) of the following options:

Test with an Alpha, Beta or Release Candidate for the next version. These can be found at http://fedoraproject.org/get-prerelease and by testing these we can work to resolve any installation issues so that the final release is free of such bugs.

However if the problem no longer exists then please close this bug or it will be closed in a few weeks if there is no additional information lodged.

get-prerelease linkThe get-prerelease link sometimes is not available or have not updated development releases.

Video Subsystem bugs

The various video subsystems that have kernel modesetting support would just prefer their bugs to be moved over to the xorg-x11-drv-{intel|nouveau|ati} component.
There, their triagers can gather needed info and debug the issue. If the problem is in the kernel, those maintainers can fix it, no need to leave the bug assigned to
the kernel.

This bug is in a video subsystem that has a kernel part. We track and work on these bugs via the driver package name instead of leaving them assigned to the kernel

Rebase Regressions

The kernel is often rebased to the latest stable kernel release and pushed as an update. This fixes a large amount of bugs, but it inevitably introduces a few regressions. If a bug is reported and it is determined that it worked with the previous kernel release, then a rebase regression entry should be added to the whiteboard field in the bug. This entry takes the form of:

rebase-regression-<kernel version>

E.g. if there is a regression when moving to the 3.1 kernel, it should be:

rebase-regression-3.1

The latest working Fedora kernel version and the earliest broken version should be noted in the comment section if possible.

This will allow the kernel developers to track regressions introduced by a rebased kernel. That will also help facilitate our communication with the upstream kernel developers, as we can report these bugs to them to be included in the regression lists that are published upstream.

Bugs with TAINTED modules

Certain bug reports refer to a kernel module that is not open source software (mark this bugs with the keyword TAINTED):

Thank you for taking the time to report this bug. Sadly, your report contains information about a not open-sourced module. This means you have loaded a module into your kernel that we can't debug, making it difficult to isolate the issue. Can you reproduce the issue without loading the tainted <module-name> module?.

If you cannot duplicate the bug without the tainted module loaded it will be closed.

However, you can report it to the vendor of your non-free module.

Stale Bugs

If a bug has been in NEEDINFO with no response from the reporter for 2 weeks, close the bug out with INSUFFICIENT_DATA and the following text:

This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 2 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.