Copr (Cool Other Package Repo) is a Fedora project to help make building and managing third party package repositories easy.

Copr (Cool Other Package Repo) is a Fedora project to help make building and managing third party package repositories easy.

+

The instance to be installed within Fedora Infrastructure provides Fedora maintainers with the ability to create repos of packages to build against and share with others.

== Some Design Requirements ==

== Some Design Requirements ==

Line 11:

Line 12:

</ul>

</ul>

−

== Where to start ==

+

== Design Overview ==

<ul>

<ul>

<li> We're creating a new build system that's more lightweight than koji to host this. the build system will also be named Copr. It will have a small web UI and a command line client. We'll create this using TG2 as it's something we're familiar with.

<li> We're creating a new build system that's more lightweight than koji to host this. the build system will also be named Copr. It will have a small web UI and a command line client. We'll create this using TG2 as it's something we're familiar with.

Line 29:

Line 30:

</ul>

</ul>

−

== Packager workflow ==

+

=== Packager workflow ===

<ul>

<ul>

<li> tool to submit an srpm to the above (using their fedora certs/account) and specify

<li> tool to submit an srpm to the above (using their fedora certs/account) and specify

Line 36:

Line 37:

<li>puts them into the repo they are being built for

<li>puts them into the repo they are being built for

</ul>

</ul>

−

== Scripts to start on ==

== Scripts to start on ==

Line 50:

Line 50:

== Design work ==

== Design work ==

−

=== KDE RedHat ===

+

−

kde-redhat is a large external repo, ask rdieter how he manages it so far.

+

+

=== Usage Cases ===

+

* Bob would like to try out a new version of GTK2. Bob does a copr build of the new GTK2 package, and then builds a number of other packages against this. Bob shares his gtk2newness copr with others for validation of the new GTK2

+

+

* Jill would like to experiment with rpm using a different compression method for packages. Jill builds a modified version of rpm for her rpm copr and builds a number of other packages with this rpm. Jill shares her rpm copr for further testing of the rpm change.

+

+

* Dave would like to experiment with kernel patches and have the repo creation handled for him. Dave continues to add patches to kernel source rpms and builds them for his copr(s). Dave shares his various kernel coprs with the community for early testing and validation. Dave spends more time working on kernel and less time dealing with repo management.

+

+

* Fedora Infrastructure would like to create a repo with updates to RHEL packages that it needs in order to manage its systems. It would like other people to be able to use the repos so that they can replicate their environment.

+

+

* The KDE SIG wants to have a repository where they can perform major upgrades that their users want but don't follow the Fedora update guidelines. By publishing to a copr repo the packages don't have to go into the main Fedora repositories but they are able to point to it for people who want to run later versions.

+

+

* The KDE SIG wants to test their updates for a Fedora release as a set so they can find regressions between the components and the release. They publish the packages to a copr repo and have their testers pull updates from there.

+

+

+

==== KDE RedHat ====

+

kde-redhat is a large external repo, asked rdieter how he manages it so far.

All I'm doing really is building stuff, either in koji or my own mock instance.

All I'm doing really is building stuff, either in koji or my own mock instance.

Line 64:

Line 80:

<li> maybe we could get some inspiration from how the opensuse buildservice works though I've only spent a couple of hours looking at it.

<li> maybe we could get some inspiration from how the opensuse buildservice works though I've only spent a couple of hours looking at it.

A Fedora maintainer would be alloted up to 1GiB of storage for package repos. Packages can be built for any arch Fedora builds for (but doesn't have to be all of them). Copr would have repoview content for easy browsing and subscribing to changes. Copr's could be increased by request.

−

== Overview ==

+

−

Koji Personal Repos (KoPeR) is a concept to allow Fedora maintainers to create personal repos of packages to build against and share with others.

+

−

+

−

=== Usage Case ===

+

−

* Bob would like to try out a new version of GTK2. Bob does a koper build of the new GTK2 package, and then builds a number of other packages against this. Bob shares his gtk2newness koper with others for validation of the new GTK2

+

−

+

−

* Jill would like to experiment with rpm using a different compression method for packages. Jill builds a modified version of rpm for her rpm koper and builds a number of other packages with this rpm. Jill shares her rpm koper for further testing of the rpm change.

+

−

+

−

* Dave would like to experiment with kernel patches and have the repo creation handled for him. Dave continues to add patches to kernel source rpms and builds them for his koper(s). Dave shares his various kernel kopers with the community for early testing and validation. Dave spends more time working on kernel and less time dealing with repo management.

+

−

+

−

* Fedora Infrastructure would like to create a repo with updates to RHEL packages that it needs in order to manage its systems. It would like other people to be able to use the repos so that they can replicate their environment.

+

−

+

−

* The KDE SIG wants to have a repository where they can perform major upgrades that their users want but don't follow the Fedora update guidelines. By publishing to a kopers repo the packages don't have to go into the main Fedora repositories but they are able to point to it for people who want to run later versions.

+

−

+

−

* The KDE SIG wants to test their updates for a Fedora release as a set so they can find regressions between the components and the release. They publish the packages to a kopers repo and have their testers pull updates from there.

+

−

+

−

=== Resources ===

+

−

A Fedora maintainer would be alloted up to 1GiB of storage for package repos. Packages can be built for any arch Fedora builds for (but doesn't have to be all of them). koper repos would have repoview content for easy browsing and subscribing to changes.

+

=== Work Flow ===

=== Work Flow ===

<ol>

<ol>

−

<li>Bob requests his account becomes koper capable.

+

<li>Bob requests a Copr repo via the Copr WebUI -- defines what other repos are dependencies

<ul>

<ul>

−

<li>this seems like a non-issue - but it may mean something for koji

+

<li>The Copr service creates an empty repo and generates an rpm that has yum repository definitions for the repo and dependencies on other repo-configuring rpms

<ul><li>first this checks the pkg and looks at the License tag to make sure it's not

+

<ul><li>first this checks the pkg and looks at the License tag to make sure it's not something we absolutely cannot build.

−

something we cannot trust.

+

<li>this just delivers the srpm to the shared filesystem and then kicks off a build task for it in headhunter.

−

<li>this just calls koji, ultimately

+

<li>Notes about building from SCM rather than SRPM; SCM server needs to be accessible from builders (currently, within Fedora's network)

−

<li>Notes about building from SCM rather than SRPM; SCM server needs to be accesible from builders (currently, within Fedora's network)

+

<li>Building from SCM is not necessarily as important as being able to share changes via the SCM. If I can clone the SCM repo and you can as well to make changes but we create an SRPM as an intermediate step for build that's not much of a burden.

−

<li>Building from SCM is not necessarily as important as being able to share changes via the SCM. if I can clone the SCM repo and you can as well to make changes but we create an SRPM as an intermediate step for build that's not much of a burden.

+

</ul>

</ul>

−

<li> Koji creates a buildroot based upon the rawhide collection and any packages in Bob's "gtknewness" koper.

+

<li> Copr creates a buildroot based upon the rawhide collection and any packages in Bob's "gtknewness" copr.

<ul>

<ul>

<li>two issues here: getting koji to use arbitrary repos

<li>two issues here: getting koji to use arbitrary repos

<li>not having the above beat the crap out of fedorapeople with koji constantly hitting it for pkgs/repodata

<li>not having the above beat the crap out of fedorapeople with koji constantly hitting it for pkgs/repodata

+

<li>These are some of the reasons we need a different build system and different storage.

</ul>

</ul>

−

<li> Koji builds Bob's gtk2

+

<li> Copr/headhunter builds Bob's gtk2

−

<li> $TOOL takes the koji built package and adds it to Bob's "gtknewness" koper, recreating yum repodata and repoview content.

+

<li> headhunter takes the koji built package and adds it to Bob's "gtknewness" copr, recreating yum repodata and repoview content.

<ul>

<ul>

−

<li> this means xferring the file back to the system or getting koji to build repodata

+

<li> this means having a task to build repodata

<li> need something in here for SIGNING those pkgs, too.

<li> need something in here for SIGNING those pkgs, too.

<ul>

<ul>

−

<li> Probably want a separate key for each kopers repo.

+

<li> Probably want a separate key for each Copr repo.

<li> why a separate key? if I'm building these for ME and mine there is no reason

<li> why a separate key? if I'm building these for ME and mine there is no reason

why I shouldn't use the same key

why I shouldn't use the same key

Line 142:

Line 140:

</ol>

</ol>

−

=== Content ===

+

== Content ==

−

Content within ones koper(s) must be content acceptable for distribution within Fedora.

+

Content within one's Copr(s) must be content acceptable for distribution within Fedora. (The packaging doesn't need to be, just the content).

To create a koper system various tools in the Fedora infrastructure would need to have changes made to them.

−

−

=== Koji ===

−

In order to build packages against a koper, koji would need to gain the ability to reference external yum repos for (at least) scratch builds.

−

−

=== fedorapeople.org ===

−

In order to support kopers, fedorapeople.org would need to be able to:

−

* Track a separate quota for kopers

−

* Accept a signal to recreate repodata and repoview content in kopers

=== Trademark Guidelines ===

=== Trademark Guidelines ===

−

We'd like to enable people to make development like spins of Fedora that include content from kopers. Since all content in kopers has to adhere to Fedora's guidelines it should be safe. We just need terms in the trademark that allow for this kind of spin creation and sharing so that these people don't have to spend time re-branding and can focus more on development.

+

We'd like to enable people to make development like spins of Fedora that include content from Coprs. Since all content in Coprs has to adhere to Fedora's guidelines it should be safe (?What do we mean by safe and what guidelines make it so? TK). We just need terms in the trademark that allow for this kind of spin creation and sharing so that these people don't have to spend time re-branding and can focus more on development.

=== New Tools ===

=== New Tools ===

−

A few new tools might be needed for dealing with kopers.

+

A few new tools might be needed for dealing with Coprs.

==== Tool to build ====

==== Tool to build ====

−

A script or make target would be needed to initiate a koper build. This is necessary to tell koji what koper to build against, as well as to get the results of the built added to said koper.

+

A script would be needed to initiate a Copr build. This will submit the build to the Copr buildsystem.

−

==== Tool to add to koper ====

+

==== Tool to add package to a copr ====

−

A tool would be needed to take the koji scratch build output and add it to ones koper, signaling fedorapeople.org to create repodata and repoview content. It should be able to replace older builds of the same package to save quota space. This tool could also be used to add already built packages to an existing koper without having to go through a koji build.

+

The build system needs to be able to add the package to the copr on the shared filesystem and create new repodata for it. We want to be able to remove older versions of packages. But we also want to have former packages so that people can downgrade around problems if they have to.

+

+

This tool could also be used to add already built packages to an existing koper without having to go through a koji build (?Do we want this? TK).

== Version 2.0 ==

== Version 2.0 ==

−

Here are some thoughts for future enhancements once we prove that we can walk.

+

Here are some thoughts for future enhancements once we prove that we can walk. Some of these may go into the first version since we have to write a new buildsystem to deal with this.

+

+

* Copr groups. Be able to have a group copr that multiple people work on. This should be made as easy as possible to get the most people as possible working together. Start with the idea of open. Extremely useful for feature driven development.

+

* Copr directory. Have a page off of the copr web ui that lists all the (active?) coprs. Easy to walk the filesystem looking for */copr/*. Read from .description files in copr directories for text to display in the directory

+

* Autogeneration of copr-repo packages for end users to easily install to use said copr. (or maybe register .repo files with pirut to process?)

+

+

== Notes from former comments ==

+

* How do we manage versioning?

+

** We don't. We can mandate a repotag suffix so we have something like fc13.copr but the packages in coprs can replace packages in Fedora so they can have versions that are more recent than Fedora. The repotag is there to help differentiate between copr builds and Fedora builds.

+

** As a side note, Coprs can very easily lead to dependency hell. PPAs have done that to Ubuntu users. Messaging about what Coprs is for and how it can lead to dependency problems is very important otherwise we'll have end users complaining about dep hell.

−

* koper groups. Be able to have a group koper that multiple people work on. This should be made as easy as possible to get the most people as possible working together. Start with the idea of open. Extremely useful for feature driven development.

+

* How do we address the concern that bugzilla bugs will be reported against copr builds rather than Fedora builds?

−

* koper directory. Have a page off of fedorapeople.org that lists all the (active?) kopers. Easy to walk the filesystem looking for */koper/*. Read from .description files in koper directories for text to display in the directory

+

** Nothing we can do about it really except ask people for their rpm -q output. We can try to message that Coprs will break your system but that's about it.

−

* Autogeneration of koper-repo packages for end users to easily install to use said koper. (or maybe register .repo files with pirut to process?)

+

−

== Comments ==

+

* What keys will we sign with?

−

If you wish to leave a comment about this concept, you can do here. I'll try to address them as I can.

+

** Something that isn't used for any Fedora release. However, it needs to be something that the new build system can sign with if we're going to enable group repositories.

−

* MatthiasClasen: The examples make it appear that this is mostly geared towards replacing Fedora packages with own builds. I'd like to see some explanation or guidance on how this will work wrt to versioning (do we require a mandatory suffix for the release, similar to disttags ?). How do we address the concern that bugs reported against such private gtk2 builds will end up in my bugzilla queue ?

+

* This would be very nice for new packages or packagers. The review process could be streamlined by running some automatic tests (such as rpmlint) on the packages as soon as they're built. Also the ability to use external yum repos would make it much easier to setup a private koji system for making derivative Fedora derived distributions. Say a company internal release or a new live CD.

−

** JesseKeating: The same things we have in place today for these kinds of things... not much (to begin with). However it really boils down to a consumer education kind of thing, educating the consumers of kopers that they're keeping all the pieces if it breaks and whatnot. The packages obviously wouldn't be signed by a Fedora key so that could be one method of discovery.

+

** It may be possible to add an rpmlint check for packages in the build system but it wouldn't be a first implementation.

−

*** [[User:till]]: Which key will be used to sign the packages or is it planned to not sign them at all?

+

−

**** [[User:jkeating]]: They could be signed with the user's key. Since a user might be able to stuff anything they want into their own repo, I wouldn't feel all that comfortable signing it with a "koji" key. That should be reserved for builds that are produced from repos that koji has ultimate control over.

+

−

* RussellHarrison: This would be very nice for new packages or packagers. The review process could be streamlined by running some automatic tests (such as rpmlint) on the packages as soon as they're built. Also the ability to use external yum repos would make it much easier to setup a private koji system for making derivative Fedora derived distributions. Say a company internal release or a new live CD.

+

[[Category: Package Maintainers]]

[[Category: Package Maintainers]]

Revision as of 15:15, 8 April 2010

Copr (Cool Other Package Repo) is a Fedora project to help make building and managing third party package repositories easy.
The instance to be installed within Fedora Infrastructure provides Fedora maintainers with the ability to create repos of packages to build against and share with others.

Some Design Requirements

Copr need their own builders to protect against badness when installing
pkgs for builddeps into buildroots

the system hosting this becomes more critical to infrastructure so it's probably best not to reuse fedorapeople for the hosting

Design Overview

We're creating a new build system that's more lightweight than koji to host this. the build system will also be named Copr. It will have a small web UI and a command line client. We'll create this using TG2 as it's something we're familiar with.

Copr will send tasks to a generic task scheduler that will drive the actual builds.

For now, the task scheduler will communicate with the builders using func. (toshio working on this)

The builders need to have a vm based mock to do the actual builds to protect against malicious packages compromising the builder for future builds (skvidal working on this)

People consuming from the Copr repos need to have a WebUI for this

Yum plugin to make configuring copr repos (with inter-repo deps) work

Hardware to test this on (mmcgrath has provided us with test hardware) and hardware to run it in production (spot when we're ready)

host for builders

host for task scheduler

host for web ui (app servers?)

shared storage for repositories, uploaded srpms, and other things all builders need access to

host for exposing the repositories to people downloading packages from copr (app servers?)

Packager workflow

tool to submit an srpm to the above (using their fedora certs/account) and specify
any targets/repos it should use

user is prompted to sign the built packages (or not)

puts them into the repo they are being built for

Scripts to start on

sync-to-repo: takes a dir of localpkgs and remote read and write path

syncs down the repodata from the remote read path and merges the localpkgs
into that path and pushes the data back up via the write path.

Design work

Usage Cases

Bob would like to try out a new version of GTK2. Bob does a copr build of the new GTK2 package, and then builds a number of other packages against this. Bob shares his gtk2newness copr with others for validation of the new GTK2

Jill would like to experiment with rpm using a different compression method for packages. Jill builds a modified version of rpm for her rpm copr and builds a number of other packages with this rpm. Jill shares her rpm copr for further testing of the rpm change.

Dave would like to experiment with kernel patches and have the repo creation handled for him. Dave continues to add patches to kernel source rpms and builds them for his copr(s). Dave shares his various kernel coprs with the community for early testing and validation. Dave spends more time working on kernel and less time dealing with repo management.

Fedora Infrastructure would like to create a repo with updates to RHEL packages that it needs in order to manage its systems. It would like other people to be able to use the repos so that they can replicate their environment.

The KDE SIG wants to have a repository where they can perform major upgrades that their users want but don't follow the Fedora update guidelines. By publishing to a copr repo the packages don't have to go into the main Fedora repositories but they are able to point to it for people who want to run later versions.

The KDE SIG wants to test their updates for a Fedora release as a set so they can find regressions between the components and the release. They publish the packages to a copr repo and have their testers pull updates from there.

KDE RedHat

kde-redhat is a large external repo, asked rdieter how he manages it so far.

All I'm doing really is building stuff, either in koji or my own mock instance.
Grabbing those rpms, signing them, stuffing it into repos.
Whatever kopers end up being, can't be much worse than what I'm doing now. honestly.

repos managable by > 1 person (ie, a sig)

custom buildroots (ie, to use itself or other kopers). we've had repo heirarchies do something like that already kde-unstable depends on kde-testing depends on kde-stable

easy mirroring, I suppose, would be nice too

not sure what kind of workflow to get packages into kopers would look like. something bodhi like, or maybe doesn't even have to be that complicated

maybe we could get some inspiration from how the opensuse buildservice works though I've only spent a couple of hours looking at it.

Resources

A Fedora maintainer would be alloted up to 1GiB of storage for package repos. Packages can be built for any arch Fedora builds for (but doesn't have to be all of them). Copr would have repoview content for easy browsing and subscribing to changes. Copr's could be increased by request.

Work Flow

Bob requests a Copr repo via the Copr WebUI -- defines what other repos are dependencies

The Copr service creates an empty repo and generates an rpm that has yum repository definitions for the repo and dependencies on other repo-configuring rpms

first this checks the pkg and looks at the License tag to make sure it's not something we absolutely cannot build.

this just delivers the srpm to the shared filesystem and then kicks off a build task for it in headhunter.

Notes about building from SCM rather than SRPM; SCM server needs to be accessible from builders (currently, within Fedora's network)

Building from SCM is not necessarily as important as being able to share changes via the SCM. If I can clone the SCM repo and you can as well to make changes but we create an SRPM as an intermediate step for build that's not much of a burden.

Copr creates a buildroot based upon the rawhide collection and any packages in Bob's "gtknewness" copr.

two issues here: getting koji to use arbitrary repos

not having the above beat the crap out of fedorapeople with koji constantly hitting it for pkgs/repodata

These are some of the reasons we need a different build system and different storage.

Copr/headhunter builds Bob's gtk2

headhunter takes the koji built package and adds it to Bob's "gtknewness" copr, recreating yum repodata and repoview content.

this means having a task to build repodata

need something in here for SIGNING those pkgs, too.

Probably want a separate key for each Copr repo.

why a separate key? if I'm building these for ME and mine there is no reason
why I shouldn't use the same key

Repos can have multiple people adding to them so there needs to be one key for it.

you could use the same key for all repositories that only you contribute to but then we'd need to keep track of two different types of repository

Bob proceeds to create srpms for various other packages and builds them in koji against his "gtknewness" koper.

having koji know the repo exists or to have it know when the repo no longer exists.

Bob shares his "gtknewness" koper with the world

this is just advertising a url

Bob creates a Desktop Live image based on rawhide and his "gtknewness" koper and shares that with the world too.

Bob fixes bugs with his gtk2 and continues to do builds until everybody is happy

Bob commits his gtk2 changes to package SCM and does a normal koji build.

Bob profits.

These last 4 items don't matter at all they are outside of scope, imo.

Content

Content within one's Copr(s) must be content acceptable for distribution within Fedora. (The packaging doesn't need to be, just the content).

Trademark Guidelines

We'd like to enable people to make development like spins of Fedora that include content from Coprs. Since all content in Coprs has to adhere to Fedora's guidelines it should be safe (?What do we mean by safe and what guidelines make it so? TK). We just need terms in the trademark that allow for this kind of spin creation and sharing so that these people don't have to spend time re-branding and can focus more on development.

New Tools

A few new tools might be needed for dealing with Coprs.

Tool to build

A script would be needed to initiate a Copr build. This will submit the build to the Copr buildsystem.

Tool to add package to a copr

The build system needs to be able to add the package to the copr on the shared filesystem and create new repodata for it. We want to be able to remove older versions of packages. But we also want to have former packages so that people can downgrade around problems if they have to.

This tool could also be used to add already built packages to an existing koper without having to go through a koji build (?Do we want this? TK).

Version 2.0

Here are some thoughts for future enhancements once we prove that we can walk. Some of these may go into the first version since we have to write a new buildsystem to deal with this.

Copr groups. Be able to have a group copr that multiple people work on. This should be made as easy as possible to get the most people as possible working together. Start with the idea of open. Extremely useful for feature driven development.

Copr directory. Have a page off of the copr web ui that lists all the (active?) coprs. Easy to walk the filesystem looking for */copr/*. Read from .description files in copr directories for text to display in the directory

Autogeneration of copr-repo packages for end users to easily install to use said copr. (or maybe register .repo files with pirut to process?)

Notes from former comments

How do we manage versioning?

We don't. We can mandate a repotag suffix so we have something like fc13.copr but the packages in coprs can replace packages in Fedora so they can have versions that are more recent than Fedora. The repotag is there to help differentiate between copr builds and Fedora builds.

As a side note, Coprs can very easily lead to dependency hell. PPAs have done that to Ubuntu users. Messaging about what Coprs is for and how it can lead to dependency problems is very important otherwise we'll have end users complaining about dep hell.

How do we address the concern that bugzilla bugs will be reported against copr builds rather than Fedora builds?

Nothing we can do about it really except ask people for their rpm -q output. We can try to message that Coprs will break your system but that's about it.

What keys will we sign with?

Something that isn't used for any Fedora release. However, it needs to be something that the new build system can sign with if we're going to enable group repositories.

This would be very nice for new packages or packagers. The review process could be streamlined by running some automatic tests (such as rpmlint) on the packages as soon as they're built. Also the ability to use external yum repos would make it much easier to setup a private koji system for making derivative Fedora derived distributions. Say a company internal release or a new live CD.

It may be possible to add an rpmlint check for packages in the build system but it wouldn't be a first implementation.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, and JBoss are trademarks or registered trademarks of
Red Hat, Inc. or its subsidiaries in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.
The Fedora Project is maintained and driven by the community and sponsored by Red Hat. This is a community
maintained site. Red Hat is not responsible for content.