1. Roadmap and Feature Planning

Purpose

To create the initial time-line for the release: the roadmap

To identify the main technical goals for the new release

Entry Criteria

E1.1 Previous release version of the distribution was released

Tasks

T1.1 Create the roadmap description. This task is done taking the last roadmap as a template and working around the holidays in China, Germany, Czech Republic and USA. A detailed description about the rules used to build the roadmap is in [15]. The final roadmap need to be written in XML using the schema from [4]. This roadmap contains all the information from T1.1 and T1.2. This XML is published using the SIMILE JavaScript program [5] to create a time line web page as the official reference. In parallel with this, the roadmap is published in the wiki from the project [1], and maintained during the release process, linking the new milestones when they are released. The important dates (roadmap and freezes) are made explicit and easy to consult for users and developers.

T1.2 Feature freeze calendar. Together with the roadmap calendar is the associated feature freeze calendar [1]. There are three stages of freezing: toolchain freeze after M3, base system freeze after M4 and full feature freeze after B1. After that the maintenance process starts. Freezes are scheduled at the end of milestone development periods to give time for development after a milestone release. They should not be placed too close to the next release as the frozen components should get a little testing before the milestone is out.

T1.3 Determine the new features. On the mailing list (as the primary channel for discussing ideas and features) and through other community communication channels the new features expected for this release are discussed and defined. Some of those features can be documented on openFATE [6] (not widely used) or the wiki [3]. The features can be decided inside the community (in this moment this is done only by a really small number of contributors) or internally in SUSE when there is a relation between SLE and openSUSE: a developer can create a feature in FATE for SLE and then add openSUSE afterward (this kind of feature is not visible outside) At this moment we do not have a strict policy on using openFATE. Some features are also decided later without planning and documentation. If the feature is implemented after the freeze calendar, an explicit approval from the Factory Maintainer is needed for the integration.

Verification

V1.1 Review the roadmap. The roadmap needs to be approved by the Product Manager and the Release Manager.

V1.2 Roadmap maintenance. Small changes in the roadmap are not communicated, just executed. But if a delay threatens the release day, an announcement is expected, and if the delay is serious we need to communicate it widely [17][18]. In every case T5 is involved, update the wiki.

Exit Criteria

X1.1 The roadmap and the freeze calendar are published in the mailing list

2. Package Development

Purpose

Develop and experiment without interfering with the devel projects and Factory packages

Entry Criteria

E2.1 The developer is creating a new package that is not in Factory or in devel projects, but has the intention to integrate the package in the distribution

E2.2 The developer wants to update a package that is currently in the distribution

E2.3 The developer wants to fix a bug in a package that is currently in the distribution, in response to a bug report in Bugzilla or another channel [19]

E2.4 There is a message from the messaging system (Hermes) [20][21] describing an action that needs to be done in a previous submit request

Tasks

T2.1 Create local package. If the package is new (does not yet exist in a devel project or in Factory), the developer creates a new package using the osc command line tool or the web interface. The creation of a package is a complex process well documented [22] and with policies which the developer must follow [23]. If the new package is stored in CPAN, Pypi or is a Ruby gem, there are scripts that help during the package creation, but in either case it is the packager who needs to submit the package.

T2.2 Branch Package. If the package currently exist in one of the devel projects and /or in Factory, the package developer branches the package from the original project to the local home. This process is similar to copying the files, but a relation between the new copy and the original package is retained (stored in some metadata files). The developer still needs to follow the general policies referenced in T2.1

T2.3 Fix / Update Package. The real work of the developer is expressed with this task. The package developer now can fix the bug, create a new feature or update the package. After the work is done, the developer submits the changes to OBS to launche the process described in "V2.2 Automatic checks complete successfully".

T2.4 Submit to Devel Project. Once the package is created or fixed/updated, and V2.2 is completed successfully, the package developer creates a submit request for the Devel Project (see Glossary to understand the relation with Devel Project and Factory) to get the package included there. If he / she doesn't know which Devel Project is related with the packages (maybe because is a new packages), the Package Developer need to ask in opensuse-factory[47] mailing list. The Factory Maintainer can create a new Devel Project in case that there is not a good match for the package, or more frequently, point to the already existing Devel Project.

T2.5 Automatic Submit. There are external scripts which update certain packages automatically. Those scripts are related to some very specific Devel Projects and are not generally used. When run they act like a Package Developer, updating the package and submitting directly to OBS (V2.2)

Verification

V2.1 Is in Factory. The work flow of creating a new package for future integration into Factory is different from the work flows for fixing bugs or updating packages that are currently in Factory. With this verification point the developer makes an explicit decision to change the path of the work flow.

V2.2 Automatic checks complete successfully. Every package needs to be committed in OBS which will build the package and run some standard tests. If compilation is successful and the package passes the tests, the developer can create a Submit Request.

Exit Criteria

X2.1 The developer creates a submit request in OBS. This submit request will be received by the Package Maintainer or the Project Maintainer of the package.

X2.2 The developer drops the change, aborting the development process.

Deliverables

D2.1 Submit Request. This document is a digital document created by the Package Developer when the package is successfully created in OBS and deemed ready by the Package Developer. This document is used by the Package Maintainer and the Project Maintainer to integrate the changes of the new version of the package in the devel project.

3. Devel Project Integration

Before a package can be integrated into Factory, it needs to belong to a devel project. A devel project is a set of related packages that share certain features. For example, in devel:language:python are stored all the packages related to the programming language Python. LibreOffice:Stable is the devel project where are stored the packages of LibreOffice updated to the last stable version.

Purpose

Provide a space where the main development (integrating feature releases) takes place.

To guarantee a certain level of quality and integration before the package is finally integrated into Factory. This integration process follows the cycle checkout – fix – commit – submit request, very similar to the process described in 2. Package Development.

Entry Criteria

E3.1 A submit request from a Package Developer to a devel project.

E3.2 A decision by the Project Maintainer to make changes to a package directly in the Devel Project

Tasks

T3.1 Integrate the submit request. The Project Maintainer can integrate the code coming from the submit request after the review is done in V1. Usually this integration is done using the OBS web interface or the osc command line tool. The Package Maintainer can only accept and integrate the submit request for his / her own packages.

T3.2 Package checkout. The Project Maintainer has two options for working with the packages part of the devel project: one is use the exit criteria and branch the package in his/her own home project (changing roles into Package Developer). The other option is checkout the package directly from the devel project and submit a change there. The former option is more secure and clear, as it avoids collisions with new submit requests. The latter one is a faster approach more adequate for projects where the number of maintainer and developers is low. To help in this process, some devel projects use extensions to the OSC command line tool. For example, the GNOME project uses the plug-in [24] "collab" [25] to allow advanced workflows.

T3.3 Fix / Update Package. The real work of the developer is expressed with this task. The Project Maintainer or the Package Maintainer can work, using as input the bugs from bugzilla [19], or the features planned for their own project.

T3.4 Submit to Factory. Once the Package or Project Maintainer considers the package(s) in his project good enough he/she creates a submit request for Factory.

Verification

V3.1 Review of the submit request. Before the integration, the Project Maintainer (or the Package Maintainer if the request is related with his / her own package) checks the request manually, to detect problems in the spec file, in the patches attached or in the compilation process. If no problems are found, the request is accepted and integration takes place (T3.1), otherwise the request is declined and Hermes sends a message to the creator of the submit request. Some devel projects have an internal review process, but there is not a shared policy by default.

V3.2 Compile in OBS. Every package is committed to OBS. OBS will build the binaries according to the spec file [26]. If the compilation is successful, the Project Maintainer can create a Submit Request and send it to Factory or continue the integration process with the rest of the packages.

V3.3 Automatic Security Review. For certain packages that use SETUID binaries or offer D-BUS services or PAM modules, an automatic security review takes place. The check is executed in the rpmlint phase of the package creation and if one of the security policies applies [27] the package can't be compiled. To lift the restriction on building the package the package maintainer must contact (open a ticket) the Security Team for review. As long as the package is not in the white list, it can't be submitted to Factory.

Exit Criteria

X3.1 The Package or Project Maintainer creates a submit request for Factory.

X3.2 The Package Maintainer decides to change role to Package Developer and branch the package in his/her home project.

Deliverables

D3.1 Submit request to Factory. A similar submit request can be created and sent to Factory. The Project Maintainer can group a set of submit requests into one single request which can be accepted or declined as a whole.

4. Review Process

The review process is only done for Factory.

Purpose

To improve the quality of the submit requests.

Potentially simplify the integration of the request into Factory.

To detect the most usual errors in: legal, technical (intro-package and inter-package) and security aspects.

Avoid the problem of accepting interdependent packages. With the review you can assure that before you commit into Factory, all the related packages meet a certain level of quality and follow certain policies, preventing Factory breaking due to missing packages if some of them are not up to the quality standards.

Entry Criteria

E4.1 A submit request is created by a Project Maintainer for Factory [28][29][30]

Tasks

Because this review process is a full verification step, this process is composed only by verification steps, not tasks

Verification

V4.1 Legal Review. The legal review is split in two parts: (1) the script Factory-auto [10] checks if the license(s) in the package have changed or not. If there is no change and the license is in the license database this script will accept the request. (2) If the package contains a change of the license, or is a new package with a license not found in the license database, the legal team has to manually check the request and accept or decline.

V4.2 Factory Auto Review. Factory-auto is a script [10] which does a shallow technical review, trying to find the most common problems usually found in requests. For example, this script checks if the associate package is currently successfully compiled in OBS. This check is fast and is used to decline the most evidently bad quality requests.

V4.3 Technical Review. The review team checks the more complicated technical aspects of the request. The Technical Reviewer will check if the specification file follows the set of predefined policies [23][26], look over quality of the code in patches and verify the content of the changelog file.

V4.4 Check Repo Review. This review is also automatic. Is executed by the check-repo script [10] and checks for typical integration problems. For example, this bot checks if certain dependencies are present in the Factory project, that the dependencies from Base:build are all self-contained in this same special project, or that the new group of request doesn't create new problematic dependency cycles in Factory.

Exit Criteria

X4.1 All the reviews are positive.

X4.1 If there is a negative review from one of the verification points, the request is declined and Hermes sends a notification to the sender of the request. Some automatic checks flag the request if it fails but do not completely reject the package. If this is the case, an action is needed from the submitter so it can be considered as an exit criteria. OBS also allows comments on the request, some of which need an action to be taken by the submitter.

Deliverables

D4.1 The review process changes the status of the request, showing at every moment what reviews are pending, accepted or declined.

Entry Criteria

E5.1 A submit request has been submitted from a devel project and has gone through the review process

E5.2 The Factory Maintainer decides to fix an issue inside Factory

Tasks

T5.1 Monitor Factory. On a periodic basis, the Factory Maintainer checks how Factory is doing, and if there are any problems (which solution is outside the scope of the Factory Maintainer role) he / she sends reminders to the correct roles, or reverts the packages that causes the problems. The Factory Maintainer uses three types of monitors: the Build Monitor showing the current stage of all process (failing packages) [31][32], the Submit Request queues [33] and the Users.

T5.2 Maintain XML files used by OBS to generate ISO images. The Release Maintainer updates and fixes the XML needed by KIWI [34] to generate the ISO images for the products. This process is not currently documented.

T5.3 Integrate the submit request. The Factory Maintainer accepts and integrates the submit request from the devel project to Factory. In order to avoid an overload of the OBS servers, the Factory Maintainer accepts the request following some basic rules based on experience (as in, not documented) [9]:

Do not accept a SR when a new build is in process.

If a leaf package is accepted, OBS will trigger an small subset of packages.

Base or core packages can lead to a lot of re-compilation in Factory. These are usually delayed until the end of the week.

T5.4 Revert or remove packages. If the maintainer of the package does not respond in time or if the package causes great pains, the change can be reverted or the package removed.

T5.5 Issue notification. The Factory Maintainer detects and notifies the Packages or Project Maintainers when there is an issue related with the package in Factory.

T5.6 Create staging projects. To simplify the integration process and issue detection the Factory Maintainer can create staging projects. A staging project usually contains a new important package (like GCC or Perl) and all the dependencies and otherwise related packages. In this way the maintainer can detect new issues related only to the new package, avoiding issues caused by other integration actions or updates taking place.

T5.7 Create Patterns. The Factory Maintainer creates and update the patterns needed for the installation [36]. The patterns format and purpose are described here [35][37]

Verification

V5.1 Is Factory open for submissions? Check for freezes or other reasons not to accept SR's

V5.2 Brief review. The review process filters low quality submit requests, so the Factory Maintainer does not verify quality. Instead he/she judges if this is the proper time to accept the request and start the build process. The Factory Maintainer usually does a simple review of the request, and can decline or put it in hold if:

The submit request is (related to) a package which is currently frozen AND the package is not fixing an issue

The package will break so many other packages that a staging project will be needed

Exit Criteria

X5.1 The package is integrated in Factory

X5.2 The package is declined and a event in reported in Hermes

X5.3 A bug report is created in bugzilla

Deliverables

D5.1 Package integrated into Factory

D5.2 A working Factory repository

6. Quality Assurance Process

The QA process can be done automatically using the openQA [11][12] tool, but also can be done manually inside a virtual machine or a real machine.

Purpose

Determine the quality of the product generated after T4. Factory Integration.

Quantify the final quality of build, detect errors and update Bugzilla with the new errors to be fixed.

Entry Criteria

E6.1 There is a new automatic build generated by OBS

E6.2 The Release Manager or the Factory Maintainer has decided to test a concrete build image, candidate as a final product (Milestone or Beta)

E6.3 The QA team has decided to test a new feature or reproduce a bug reported in BNC [19]

Tasks

T6.1 Create a new test. If the QA Team decides that the current coverage of the test set is not enough, they need to write a test that covers the new feature / use case.

T6.2 Test maintenance. The way to make tests for Factory is using openQA. This tool requires the maintenance of certain components named needles, used in the tests to assert outcome conditions [46]. These needles need to be updated and adjusted when an important change in the artwork or other areas effecting the look and behavior of the distribution occurs.

T6.3 Create bug reports. openQA detects problems in the distribution, but it does not specify the exact issue which affects the ISO image. The QA Reviewer will dig into the failed test, deduce what is wrong and document the findings in a bug report published in bugzilla. This bug is assigned to the correct developer to fix the bug in the T2. Package Development task or in T3. Devel Project Integration.

Verification

V6.1 Test review. The tests are grouped according to small set of criteria: the phase of the test (installation process, text applications or graphical applications) and the relative importance (critical, important, normal). The tool can mark one ISO as failed if critical tests are not successful passed, or if certain tests fail in the installation process. If the final result in openQA is not green the ISO has to be debugged and a bug report be written (T6.3)

Exit Criteria

X6.1 The important and critical test are green and the ISO is considered candidate for a Milestones or Beta release

X6.2 An important bug is detected by the tool in some configuration and the ISO is rejected as a viable product candidate.

Deliverables

D6.1 Test report. The application generates a test report with screen-shots and status of the test. This report is reviewed by the QA Reviewer.

D6.2 Bug report. If a bug is found (critical or not), the QA Reviewer creates a bug report with the description of the bug and assigns it to the correct maintainer of the package.

T7.5 Create list of changes for marketing. Release manager generates differences between current and previous milestone with dvddiff script and sends it to marketing together with other important notes. The resulting list is shortened to a more manageable number (between 10 and 30) based on the importance of the updates.

T7.6 Branch sources for Beta. In case of Beta, release manager branches all the sources in Factory off. When the branch is made, Factory is open for development again.

T7.7 Create announcement. Marketing manager creates announcement based on diff and the template on progress.o.o [40]

T7.8 Sync image to mirrors. Release manager syncs the image and binary repository to the mirrors [41]. Release manager notifies mirror admins about the new iso's and the proposed release date by mailing mirror@opensuse.org. Release manager publishes the image on the mirrors (making files readable)

Verification

V7.1 Pick image. Release manager decides, based on openQA output, that a potential release image is good enough. If the release is an early milestone, the Release Manager can decides to release using a different criteria than openQA output. The closer to the release, the more the Release Manager raises the threshold for releasing.

V7.2 Discuss roadmap for Beta. If this is Beta, a meeting with the product manager and Management is set to discuss the state of Factory and the roadmap. There is a go / no-go meeting for the final release with the Release Manager, Controller, QA Reviewer, the Product Manager and management where it is decided to delay or not.

V7.3 Check communication. The Release Manager checks if the delay (if any) is big enough that it has to be communicated

V7.4 Check Sync state. The Release Manager checks if the image is synced and published to the mirrors

Exit Criteria

X7.1 Release is available and announced

Deliverables

D7.1 ISO images for all supported platforms and architectures

D7.2 Release announcement text for news.o.o and mailing lists

D7.3 In case of delay: announcement and new roadmap

8. Public Test

After the release process, a second QA is done by the community (developers and users). This test is not automatic and is usually done on real hardware

Purpose

Determine the quality of the product generated after the release

Identify real bugs in real hardware and in real use case

Detect errors and update Bugzilla with the new errors to be fixed.

Entry Criteria

E8.1 A new release (Milestone or Beta) is published on sofware.o.o [14]

Tasks

We can consider this action as a task or as a verification point. If this is a verification point, there are no real task assigned to this action.

Verification

V8.1 Test in real machine. The user (or developer) can go to software.o.o [14] to download the release and install it in the real hardware. If he / she detect a bug a new bug report is created in bugzilla [19] or / and a mail is send to the Factory mailing list. Also, many users decide to test Factory directly updating the repositories [50], without installing the released media.

Exit Criteria

X8.1 A new release is published

Deliverables

D8.1 Bug report. If a bug is found, the tester creates a bug report with the description of the bug. The QA Reviewer can reproduce the bug and assign the it to the correct developer.

Project Organization

1. Roles and Responsibilities

There is a large number of people involved in Factory development. We have identified the following roles:

Some of these have one or more roles assigned in the Open Build Service. See an overview [42][43][44][45].

Glossary

Branch

When a contributor wants to work on a package, the first step is to create a copy of the original package. OBS will establish a relation between the original package and the copied (branched packaged). Both packages can evolve separately until a merge process occurs (the integration process).

BNC

See Bugzilla

Bugzilla

Management tool used to collect, monitor, assign and update bug reports. The project is using the bugzilla installation from Novell [19]. This site is also known as BNC.

Commit

When a developer (contributor) is working on a package, he / she needs to register the changes made. This registration process is known as committing, and is done in OBS using the OSC command line tool or automatically in the web interface.

Devel Project

Project which is a upstream part of Factory. Factory is composed with a set of different Devel Projects that are integrated to build the next release version. In [48] there is more formal description of what is a Devel Project. A Devel Project can have one or more maintainers who decides the features (roadmap) implemented for this package and work to integrate them in the project.

ETVX

Formal methodology to describe a process developed by IBM in 1985. This methodology is classified as narrative, instead of graphical [13] (flowcharts).

Factory

Factory is a project stored in OBS that integrates the different Devel Projects with the goal of creating a future openSUSE release. The work in Factory is usually done by the Factory Maintainers (integrate the different project packages) and the Release Managers (create releases based on ISO images from Factory).

Feature Freeze

To make the integration of the devel project into Factory easier, the Release Manager establishes a calendar with periods during which certain packages are now allowed to be updated (are frozen). Only bug fixes (and no new versions or new features) are allowed until the Release Manager and the Factory Maintainer start a new release cycle. Exceptions can be made.

Hermes

Hermes [21] is the central notification hub of the openSUSE Project. It gives users control over the kind of notifications they want to receive, in which way and when.

Home Project

Every developer has a home project assigned which can be used to develop and experiment without interfering with the rest of the packages stored in the different projects. When a developer branches a package from a devel project or from Factory, the copied package is stored automatically in his or her home project.

KIWI

The openSUSE KIWI [34] Image System provides a complete operating system image solution for Linux supported hardware platforms as well as for virtualisation systems like Xen, Qemu or Vmware.

OBS

The Open Build Service (OBS) [8] is a generic system to build and distribute binary packages from sources in an automatic, consistent and reproducible way. Is used to release packages as well as updates, add-ons, appliances and entire distributions for a wide range of operating systems and hardware architectures.

openFATE

openFATE [6][7] is the Feature- and Requirements Management System of the openSUSE community. Used to open the process of product planning to all openSUSE members.

openQA

With openQA [11] we can automatically run tests over a distribution. Some tests are designed to make a full install of the distribution in a virtual (or real) machine, with different configurations of file systems, desktop environment, architectures, medias, etc.

OSC

Command line tool used to interact with OBS

Package

A source package is a list of files (usually a tar file and resources like patch files) and a specification file that can be used to generate one or more binary packages

Pattern

A predefined set of packages related with a specified configuration or functionality. Patterns are used to install a set of packages that can form a usual configuration (laptop, server, desktop, developers, office user, etc.)

A plan that matches short-term and long-term goals with specific technology solutions to help meet those goals

Staging Project

Temporal projects used to test critical components before they are submitted to Factory. Staging Projects can be created and destroyed on demand by the Factory Maintainer, but some of them, like [49] are more general and are reused frequently.

Submit Request

Digital document established by a developer and managed by OBS that describe the set of changes done for a source package.