In the interest of keeping Fedora as a self-hosted distribution (meaning you can use Fedora version Z to build Fedora version Z from source RPMs), [[User:Mdomsch|Matt Domsch]] regularly runs a full rebuild of the "rawhide" tree, building rawhide with rawhide. This catches a number of packages that no longer build, and need developer attention. The results of each run are presently mailed to each failing package's owner and cc: list (as noted in the package database), and sent to fedora-devel-list.

In the interest of keeping Fedora as a self-hosted distribution (meaning you can use Fedora version Z to build Fedora version Z from source RPMs), [[User:Mdomsch|Matt Domsch]] regularly runs a full rebuild of the "rawhide" tree, building rawhide with rawhide. This catches a number of packages that no longer build, and need developer attention. The results of each run are presently mailed to each failing package's owner and cc: list (as noted in the package database), and sent to fedora-devel-list.

−

In the interest of tracking these failures, new bugs for each failing package will be filed in Bugzilla. These bugs will all block a blocker bug, alias "FTBFS". Included in these bugs will be the root.log and build.log from mock for architectures where the failures occurred. These bugs are filed as NEW because that's all the XMLRPC interface will allow at present. They should be filed as ASSIGNED, since they are by definition pre-triaged.

+

In the interest of tracking these failures, new bugs for each failing package will be filed in Bugzilla. These bugs will all block a blocker bug, one per OS version, such as alias "F10FTBFS". Included in these bugs will be the root.log, build.log, and mock.log from mock for architectures where the failures occurred. These bugs are automatically tagged with keyword 'Triaged', per the [[BugZappers/BugStatusWorkFlow|BugZappers workflow]].

A recursive check will be made that there is not already a bug that's blocking FTBFS for the package in question. If there's not, a new bug will be filed against the package.

A recursive check will be made that there is not already a bug that's blocking FTBFS for the package in question. If there's not, a new bug will be filed against the package.

−

=== Challenges ===

+

== Challenges ==

* Avoiding false positives: It somewhat often happens that a whole class of failures are due to either build system misconfiguration, or mirrors being slightly out of sync.

* Avoiding false positives: It somewhat often happens that a whole class of failures are due to either build system misconfiguration, or mirrors being slightly out of sync.

* Bugs in required packages: If glibc is broken on a particular day, it can affect a large number of package builds. It's most appropriate to file a single bug against glibc in this case rather than many bugs against each package that hit the bug. Unfortunately, figuring this out requires human intervention.

* Bugs in required packages: If glibc is broken on a particular day, it can affect a large number of package builds. It's most appropriate to file a single bug against glibc in this case rather than many bugs against each package that hit the bug. Unfortunately, figuring this out requires human intervention.

−

1. Being that this is a monthly event, I think that simple sanity checking is really all that's required here - nothing fancy. Rebuild and filing should be two separate phases, so that these issues can be caught.

+

# Being that this is a monthly event, I think that simple sanity checking is really all that's required here - nothing fancy. Rebuild and filing should be two separate phases, so that these issues can be caught.

−

+

−

=== Process ===

+

+

== Process ==

* File bugs, once for each package.

* File bugs, once for each package.

* Block FTBFS

* Block FTBFS

* FTBFS blocks Target tracker for next release

* FTBFS blocks Target tracker for next release

−

* attach root.log and build.log from each architecture that has failed.

+

* Attach root.log and build.log, and mock.log from each architecture that has failed.

# Find an existing bug for that package describing the problem. Set your bug to "Depends on" that other bug. Do not change the component of your bug to the other package, or you will get more FTBFS bugs created against you.

# Find an existing bug for that package describing the problem. Set your bug to "Depends on" that other bug. Do not change the component of your bug to the other package, or you will get more FTBFS bugs created against you.

# When that other bug is closed, you'll get an email from bugzilla as usual. Rebuild your package using a koji scratch build, to verify it builds cleanly again. Close your bug as CLOSED, RAWHIDE, mentioning this.

# When that other bug is closed, you'll get an email from bugzilla as usual. Rebuild your package using a koji scratch build, to verify it builds cleanly again. Close your bug as CLOSED, RAWHIDE, mentioning this.

−

* If the package should be retired, see [[PackageMaintainers/RetiredPackages|Retired Packages]]

+

* If the package should be retired, see [[Retired packages]]

In all cases, if you close an FTBFS bug as a duplicate of another bug, please make the other bug to block 'FTBFS' (bug number 440169). In this way the bug left open will appear in the FTBFS reports properly.

In all cases, if you close an FTBFS bug as a duplicate of another bug, please make the other bug to block 'FTBFS' (bug number 440169). In this way the bug left open will appear in the FTBFS reports properly.

+

+

== Package Removal for Long-standing FTBFS bugs ==

+

+

Packages with unresolved FTBFS bugs at Feature Freeze for the N+2 release will be removed from the distribution. Package owners may request that their package _not_ be removed provided they are actively working on resolving the FTBFS and have a plan to resolve the FTBFS before the N+2 Alpha release. See [[Deprecate FTBFS packages]] for more details.

Revision as of 01:08, 28 November 2012

In the interest of keeping Fedora as a self-hosted distribution (meaning you can use Fedora version Z to build Fedora version Z from source RPMs), Matt Domsch regularly runs a full rebuild of the "rawhide" tree, building rawhide with rawhide. This catches a number of packages that no longer build, and need developer attention. The results of each run are presently mailed to each failing package's owner and cc: list (as noted in the package database), and sent to fedora-devel-list.

In the interest of tracking these failures, new bugs for each failing package will be filed in Bugzilla. These bugs will all block a blocker bug, one per OS version, such as alias "F10FTBFS". Included in these bugs will be the root.log, build.log, and mock.log from mock for architectures where the failures occurred. These bugs are automatically tagged with keyword 'Triaged', per the BugZappers workflow.

A recursive check will be made that there is not already a bug that's blocking FTBFS for the package in question. If there's not, a new bug will be filed against the package.

Contents

Challenges

Avoiding false positives: It somewhat often happens that a whole class of failures are due to either build system misconfiguration, or mirrors being slightly out of sync.

Bugs in required packages: If glibc is broken on a particular day, it can affect a large number of package builds. It's most appropriate to file a single bug against glibc in this case rather than many bugs against each package that hit the bug. Unfortunately, figuring this out requires human intervention.

Being that this is a monthly event, I think that simple sanity checking is really all that's required here - nothing fancy. Rebuild and filing should be two separate phases, so that these issues can be caught.

Process

File bugs, once for each package.

Block FTBFS

FTBFS blocks Target tracker for next release

Attach root.log and build.log, and mock.log from each architecture that has failed.

What to do if you get a FTBFS bug?

Commit the changes and build the package. The fixed package will land in rawhide, generally the next day.

If the build succeeds, close the bug as CLOSED: RAWHIDE, and include the package version number in the comments.

If the bug is in a different package:

Find an existing bug for that package describing the problem. Set your bug to "Depends on" that other bug. Do not change the component of your bug to the other package, or you will get more FTBFS bugs created against you.

When that other bug is closed, you'll get an email from bugzilla as usual. Rebuild your package using a koji scratch build, to verify it builds cleanly again. Close your bug as CLOSED, RAWHIDE, mentioning this.

In all cases, if you close an FTBFS bug as a duplicate of another bug, please make the other bug to block 'FTBFS' (bug number 440169). In this way the bug left open will appear in the FTBFS reports properly.

Package Removal for Long-standing FTBFS bugs

Packages with unresolved FTBFS bugs at Feature Freeze for the N+2 release will be removed from the distribution. Package owners may request that their package _not_ be removed provided they are actively working on resolving the FTBFS and have a plan to resolve the FTBFS before the N+2 Alpha release. See Deprecate FTBFS packages for more details.