April 2/2012 - Phase I beta-testing for projects.eclipse.org and PolarSys. Support for maintaining and disseminating project and release information, project proposals, and committer and project lead elections.

Revision as of 15:43, 18 March 2012

Note that this document is not intended to be all inclusive. The combination of this document, the existing Developer Portal Use Cases, and the Eclipse Development Process form a more complete picture.

Problem Statement

Current infrastructure (i.e. The Developer Portal) is inadequate.

Data concerning projects, people, organizations, members, and more is spread across multiple, separate data sources. This makes querying across data challenging. In some cases, data is replicated from one database to another to facilitate some queries while providing an added layer of protection. The database that contains committer information, the so-called "Foundation" database, is kept separate from other systems to provide an added layer of protection against private data being compromised. Bits of that database are replicated into an "Eclipse" database for access from the public website.

Portal is separate from the the resources being managed; this requires a context switch to use. The content displayed in the project summary pages, for example, comes from the portal. When a committer notices that the data in the project summary is incorrect, they must switch into the portal to make the change. Most committers have difficulty (or outright refuse) to make that context switch.

Some management tasks are spread out. Specifying a description for a project, for example, requires that an HTML file be created in the project directory (requires CVS check-in), and then the specification of a URL in the portal. This is very difficult to maintain as it is separated from where and how the description is used, and requires multiple steps with multiple tools to complete. As a result, descriptions tend to be poorly specified, and rarely maintained.

Too much information is not included in or managed by the portal. Project proposals, review documentation, IP logs, are all separate.

Technology Choices

Project management is essentially a document-management and workflow problem. Several solutions exist in this area.

The Eclipse Foundation currently uses Drupal for Eclipse Marketplace, Eclipse Live, and the EclipseCon Website. Several Eclipse Foundation employees are already well-versed in Drupal development, and finding temporary resources with the necessary skills in the local area should be relatively easy and cost-effective. Drupal is based on PHP, a language that is known to most of the Eclipse Foundation staff, and is currently in wide deployment by the Eclipse Foundation.

Perhaps one of the features that weighs most heavily in Drupal's favour is the size of the community behind it (which measures in the hundreds of thousands) and the hundreds (perhaps thousands) of plugins that are available to extend it. The availability of plugins, combined with the relative ease with which Drupal can be extended means that the overall amount of custom code that needs to be maintained should be relatively small (as compared to other solutions that may require more customization).

There are several other options that have been considered, including a handful of Eclipse-based solutions (which would allow us to "eat our own dogfood"). After careful consideration, however, we have determined that we do not have the resources to implement these solutions.

Technology comparison

Pros

Cons

Drupal

Skilled resources within existing Eclipse Foundation staff

Large number of skilled resources available

Very large community of developers, adopters, and users.

Deployed by 1,000s of organizations

Hundreds of plug-ins available to leverage in favour of writing custom code

Assumed to be no skilled Apricot developers available in the local area.

Ramp up time for new developers is relatively long.

Good quality Java-savvy developers are relatively difficult to find.

Availability and usefulness of extensions is unknown

No existing IT Infrastructure support

Roles

EMO(ED) - EMO Executive Directory (i.e. Mike)

EMO(PM) - EMO Project Manager (i.e. Wayne)

EMO(LC) - EMO Legal Council (i.e. Janet)

EMO(IP) - EMO Intellectual Property Team

Proposer

Committer

Communication Channels

The communication channels used by the system are expected to change with time as technologies and communities evolve. Favoured technologies of the day are Twitter, RSS, and the eclipse.proposals forum.

Technology

Drupal

Drupal 7.2 is used for initial implementation.

LDAP

The new system will use LDAP for user authentication and for some aspects of authorization. This implementation will be based on work being done in bug 364605 that intends to standardize the entire eclipse.org infrastructure on the LDAP (Bugzilla-based authentication is used for most eclipse.org properties; LDAP is currently only used for committer authentication).

LDAP groups are used for project-based authorization. Users who have committer access to a project are members of an LDAP group that corresponds to the project. This group membership is bound to Drupal authentication via Drupal "organic groups" functionality. Additional authorization schemes are required to handle PMC-access to projects and other corner cases. For more information, see bug 363614.

Themes

The following themes will guide the development of the new Project Management Infrastructure:

Trust committers to do the right thing;

Once and only once (avoid repeating information);

Everything is undoable (i.e. revision tracking); and

Do the simplest thing that can possibly work (avoid complexity)

We assume that committers will always endeavour to do the right thing. Specifically, we give committers leeway to do what they think is right. This, in combination with revision tracking means that mistakes can be easily undone. A committer may, for example, make ill-informed changes to a project's scope that is later reversed by the project lead or PMC.

It would be easy to get mired in multiple layers of approvals for various activities, or complex logic to prevent committers from shooting themselves in the proverbial foot. But such complexity makes the cost of later change and ongoing maintenance prohibitively high.

Note that we make a distinction between committers and general users. We do not--as a general rule--trust the general population to do the right thing. For completeness, we include with the definition of "committers" all those users with roles that permit them access to the project (including project leads, pmc members, EMO staff, etc.)

Requirements and Implementation

Requirements for the new Project Management Infrastructure are detailed in Eclipse Bugzilla (note that requirements are still be added at this time). Eclipse Bugzilla bug 363868, Reimplement the Project Management Infrastructure/Developer Portal, provides the root for the initial effort; all features and functionality planned for the initial release are captured as subtasks.

Overview and Design

Project Management Infrastructure Model

Much of the design is captured in Bugzilla (as indicated above).

The model of the system is captured in an Ecore file. Note that the model will continue to evolve as we progress through implementation.

Project

Much of the project management infrastructure revolves around the Project type. Much of the information tracked by a project is represented very simply as fields (e.g. websiteUrl and wikiUrl).

Representation of projects in the new system is not all that different from the old system. The main difference is that we will represent the project description, scope, and retention policy directly with the project.

Project information can be viewed by anybody;

Project information can be modified by any project member;

Values for all URLs represented by a project must point to eclipse.org properties (www, downloads, wiki) unless otherwise specified;

The project's relationship with Bugzilla is specified as a combination of Bugzilla Product and Component names that can be used to query Bugzilla in arbitrary ways, or provide assistance to a user in creating a bug;

Content is expected to be concise, so there should be no need for heading ("Hn") or structural ("div") tags.

Description is a short (1-2 paragraph) description of the project that is suitable for displaying on the project summary, on pages that provide an overview of multiple projects, and other places (a consistent, paragraph-based format will make for a very consistent experience);

Additional information about the project (over and above the information provided in the description) can be obtained from the project web site (websitesUrl);

Scope is the scope of the project as defined in the project proposal;

The scope is editable by anybody with access with the caveat that changes to the text are permitted, but changes to the actual meaning of the scope must be vetted by the community via Restructuring Review;

The retention policy is the formal policy set by the project regarding how they distribute their code, how older versions are retained, archived, etc.; and

Projects are marked active immediately upon creation, they are marked inactive when terminated.

Issues to be addressed:

Links to external content;

Incubation conforming vs. nonconforming;

Person

Person instances have relationships with projects and organizations via roles

Person is the generalized term that we use to represent committers, project leads, company representatives, and more.

Very basic information, including a committerId (UNIX name), firstName, and lastName are represented in the instance. A person has zero or more phone numbers, mailing addresses, and email addresses.

Instances of person represent relationships with other types through direct and indirect means. A person optionally tracks a primary employer (primaryEmployer). Indirectly, an instance has relationships with zero or more Project instances, and zero or more Organization instances; those relationships are represented via Role instances.

Roles define the relationships between a person and zero or more organizations, and zero or more projects;

Roles have a start and end date (an open end date implies that the person is currently active in the role);

Additional role types can be added;

A person has zero or more organization roles, e.g.:

Employee;

Accounting contact;

Administrative contact;

Architecture Council Representative (Strategic Members only);

Planning Council Representative (Strategic Members only);

Board representative (Strategic Members only);

Committer member;

Company representative;

Contact person for 24/7 IT support;

Delegate;

IT contact;

Legal contact;

Marketing contact;

Membership page editor; and

Organization alternate display.

A person has zero or more project roles, e.g.:

Committer;

Committer Emeritus;

Project lead;

PMC Lead (top-level projects only);

PMC Member (top-level projects only);

Architecture Council Representative (top-level projects only); and

Planning Council Representative (top-level projects only).

Elections

Elections are connected directly to the role.

In the case of a person assuming a committer, project lead, or PMC Member roles, an election is required. Elections are attached directly to the project role.

The nominee must already have an account;

An election can only be initiated by any existing project member (committer, project lead, PMC Member, or PMC Lead);

A user can only be nominated into a role if they do not currently hold that role;

The nomination criteria must specify suitable merit;

We can generate a reasonable initial stab at a of merit based on activity in Bugzilla/Git;

All project members are invited via email to vote;

Project members vote yes (+1), no (-1), or abstain (0);

The vote ends when:

all eligible voters have cast their votes; or

a full week has passed.

Structure

The application code is laid out in a directory structure containing several custom modules (details TBD). The system can be manifested directly from these modules. Modules are self-contained, are self-configuring, and self-updatable.

An installation profile for projects.eclipse.org automates the initial installation and configuration of the system, along with bootstrap data for eclipse.org projects.

Application Lifecycle Management

The Project Management Infrastructure is not itself an Eclipse project.

While under development, the system is deployed in two different configurations: one for eclipse.org and one for Polarsys. A major design point is to permit multiple separate deployments with customization ("Foundation in a box").

Source Control

Drupal presents some interesting source control management issues; it is possible to "code" much of a Drupal-based site without ever writing actual code. Many of the development tools provided for Drupal facilitate the construction of elements that exist directly in the database. When those tools are used--such as the Content Construction Toolkit (CCK)--the results are converted into code and included in the code tree. The code tree is committed into Git.

The main Git repository is currently private to employees of the Eclipse Foundation. This will change as development progresses.

Build and Release

Releases are tagged using standard three-part Eclipse version numbering strategy. The first number, the major version is updated for significant milestones (or when significant API breakage occurs); the second number, or minor version is updated with each release that adds or changes functionality; and the third number, or service version is updated for bugfix or service releases.

Builds are generated by tagging and then extracting (via git archive) the state of the system. We need to visit this decision as we do not actually use these builds as part of our deployment.

Deployment

Deploy first to staging servers for testing;

When tests pass, deploy to production servers;

Deploy directly from git (i.e. git pull);

Deployment to both projects.eclipse.org and polarsys.org

Schedule

April 2/2012 - Phase I beta-testing for projects.eclipse.org and PolarSys. Support for maintaining and disseminating project and release information, project proposals, and committer and project lead elections.