The development of the E3 collection is designed as a six-step process. These steps-discovery, evaluation, entry development, curricular review, final-draft approval, and incorporation-are outlined below. The key quality control steps are: Step 2, Content Evaluation, when the Topic Advisors working at the level of their sub-disciplines organize and evaluate content; Step 4, Curricular Review, when the content is incorporated into the context of applications for different disciplines and grade levels; and Step 5, Final Draft, when the results of step 4 are reviewed by the Editor-in-Chief.

To assist in the initial stages of content discovery, the E3 project staff has created a list of expected encyclopedia entries. This list includes several hundred terms, a mastery of which is deemed important for a complete understanding of the earthquake phenomenon and its effect on human activities. As content is discovered, it may be incorporated within one of these pre-designated entries or it may necessitate the creation of an entirely new entry. The process for collection development and review outlined below accommodates either scenario.

Step 1: Content Discovery

The initial phase of content discovery will be largely dependent upon the work of individual members of the three participating organizations. From this pool of hundreds of geoscience and engineering professionals, we expect to draw a large volume of content spanning the full scope of the collection. By raising internal awareness of E3 through mailings, direct contact with members, and special sessions at institutional meetings, we will encourage those with existing resources to take advantage of the free on-line publication of those materials, and the subsequent recognition that their publication will merit.

While we are confident that the combined resources of the three institutions participating in E3 will provide a large base of content upon which to build the initial structure of the encyclopedia, we recognize that much of the work involved in content discovery must be distributed among the community. Specifically, the initial metadata cataloging process can be extremely labor intensive; DLESE has learned that in the production of a digital resource, 2/3 of the cost is creating metadata (Metadata Focus Group PowerPoint presentation, DLESE Leadership Workshop, 30 June 2000). Thanks to our collaboration with DLESE, we will save considerable effort in the process of content discovery and metadata entry by inviting the community to contribute resources to the encyclopedia by means of two web-based systems already created by DLESE. The simpler of these two("Suggest a URL") will be aimed at casual users to solicit suggestions for resources that would then be investigated at cataloged (if appropriate) by the E3 project staff. The more advanced discovery system would be a modified version of the DLESE Resource Cataloger, a system that automatically converts the input from a web-based form (a questionnaire about specific characteristics of a resource) into an IMS metadata record and places that record into a database on a preliminary basis.

Because the scope of E3 is more focused than that of DLESE, we will effect changes and additions to the DLESE Resource Cataloger and develop metadata extensions that meet the needs of our project. These extensions will allow a refinement of searches not currently possible on DLESE as a whole, owing to the broad nature of their collection. For example, the wide range of disciplines covered in Earth system education places a limitation on the specificity of their controlled vocabularies. To address this difference in scope, we will create our own analogous controlled vocabulary lists for our E3 Resource Cataloger to assist contributors of content in assigning resources to particular areas within the fields of earthquake-related science and engineering. This, in turn, will assist the Topic Advisors as they determine the proper entry placement of content in Step 2.

The work of the voluntary contributors from our participating organizations will be done without cost to the project. We do intend to provide incentives, however. Contributors who are not creators would receive a public note of thanks on a special page of our website; creators will have their material published with appropriate credit in a widely accessible electronic publication. In some cases, a creator's material might be enhanced, and the creator would, of course, be able to reuse his or her work with these new modifications. For example, if a static map is provided and is then converted into an active map that the user can click on to access geographically-related content, that on-line product will also be made available to its originator, so that this enhanced material can be mounted on his or her website.

Step 2: Content Evaluation

The on-line E3 Resource Cataloger and "Suggest a URL", in combination with active solicitations to our institutions, are expected to generate a large quantity of potential content from contributors. If enough information about a resource has been provided to create a record within our database, the Topic Advisors will investigate and evaluate this content for possible inclusion in E3. In their evaluations, five considerations will be important:

 Relevance. Is the resource relevant to earthquake education?

 Integrity. Are there no blatant errors of fact in the resource? Are there no blatant political, religious, or commercial messages in the resource? Does it function reasonably; i.e., seem to be basically bug-free? These first two considerations are the same as the two primary filters applied to resources suggested for inclusion within the DLESE collection.

 Access. Is this content easy accessible for potential users? Are there technical, linguistical, or copyright issues which might restrict access for many users?

 Classification. Under which entry heading will this content fit, or does it require the creation of an entirely new entry in E3? If the content was submitted through the E3 Resource Cataloger, has it been properly cataloged by the contributor?

 Quality. E3 Topic Advisors will judge the quality of resources using the seven criteria DLESE has developed to select high-quality resources from their Broad Collection for promotion to their more prestigious Reviewed Collection. According the DLESE standards, high-quality resources are (a) easy to use, (b) well-documented, (c) scientifically accurate, (d) significant, (e) motivational, (f) robust and sustainable as digital resources, and (g) pedagogically effective.

The Managing Editors will be responsible for overseeing this task and managing the work of the Panels. It is expected that coordination and communication among the Topic Advisors and each Managing Editor can occur remotely without the need for face-to-face meetings, other than those provided by the partner organizations (e.g., annual meetings and research workshops) and at the special Curricula Review workshops described below.

Step 3: Entry Development

Having determined whether newly approved, raw content is part of a particular, pre-existing entry or warrants the creation of a new entry, the Topic Advisors will then assign that content to a specific place within E3 on a provisional basis (similar to, but distinct from, that of the initial, contributor-cataloged metadata record). Items marked with this provisional status will be password-accessible for reviewers as required for Step 4, but not visible to public browsers. If a new entry is needed, the process for its construction is as follows:

 The new entry term is identified by the Panel;

 This term is defined and the contents of the entry outlined (Glossary & Outline level);

 A synopsis of the entry is developed (Synopsis level);

 Curricular Connections and Content In Depth are identified and linked;

 Other search parameters are assigned;

 Any necessary permission for use of the material is obtained and the exact credit wording is established.

For content that fits within an established entry, the process will be similar, though the entry definition, outline, synopsis, and curricular connections will be modified only as needed to incorporate the new content. If necessary, these changes will be held within a revised copy of the entry visible only to reviewers. The database structure of the collection makes it relatively easy to add new content within existing entries. These draft materials will then await further curricular review and final approval before they are fully incorporated into E3.

Step 4: Curricular Connection Review

Once the Review Panels have identified and evaluated a significant set of resources for curricular use, the K-12 and college curricular reviewers will be convened at a small workshop. At this point, we will know which contributors have been the most helpful and interested in the project, and can allocate travel funds accordingly. Each workshop will deal with one of the four engineering or six geoscience topics. We should be able to schedule the four engineering workshops, each a half-day in duration, to occur consecutively over a two-day time span, in order to maximize participation and minimize travel costs. Similarly, at least three of the six geociences workshops can be scheduled together. In addition to the half-day of time allotted for the workshop, participants will be expected to devote additional time to reviewing the collection and making comments via an on-line input form. The workshop participants are shown below:

Curricula Review Workshops

Participants Unique to a Given Topical Workshop (5 to 7)
Panelist, plus his or her student (2); sometimes one or more other Topic Advisors depending upon the identified overlap in content
Two or three major contributors (3)

Participants Common to All Workshops (15)
Editor-in-Chief (1)
Managing Editor (1)
Senior Advisor (1)
Members of the K-12 community:
K-8 instructor and 2 students
High school A: 2 instructors, each with 2 students
High school B: 2 instructors, each with 2 students
SCEC and CUREE web developers (2)

Step 5: Final Draft

Depending on the recommendations of the review panels and the results of each workshop for a given set of material, one of three outcomes will be determined by the Editor-in-Chief:

 The content may be rejected if the Editor-in-Chief feels it does not truly meet the standards for relevance, integrity and access, or is otherwise deemed sub-standard in a way that cannot be remedied by reasonably simple revision.

 The content may require some modification to overcome limitations.

 The content may be accepted as-is, without need for revision.

In the case of the first outcome, the Editor-in-Chief may decide to send a notice to the creator of the material explaining the reasons why their resource did not gain approval and suggesting the kinds of modifications necessary for incorporation into the collection. The resource creator would then have the option to undertake those modifications and re-submit the material to E3. In the case of the second outcome, similar notification might also occur, though minor revisions or work-around solutions could be implemented by the E3 project staff with the approval of the creator. Once these revisions are completed, the material would gain final approval from the Editor-in-Chief.

Step 6: Incorporation Into The Collection

Content that has received final approval from the Editor-in-Chief will be incorporated into the collection by the SCEC web developer. This content may take the form of a new entry in E3 or be integrated as part of a pre-existing entry. Maintenance of this content-for example, the checking (and if necessary, re-establishment or removal) of links-is an extension of this step. Should an external resource degrade in such a way that it no longer meets the standards of E3 or serves the needs of the digital library community, the SCEC web developer will bring this to the attention of the Editor-in-Chief, who will decide if that resource should be removed from the collection.

An additional and ongoing part of collection development and review for E3 will occur once new content has been approved and mounted on-line. We will create a simple, user-friendly option for quickly rating the usefulness of resources contained within the collection and providing written reviews. These ratings and reviews will be stored within the metadata of that resource, updated when appropriate, and made available as an option in user search results. This system will give users the chance to make their voices heard and become part of the active digital library community. It will also provide collaborators with a peer-based quality check to assist them in further refining their materials. The user-review system will also provide a real-world assessment of the types or styles of resources most highly valued by students and educators. In a similar way, user reviews might ultimately begin to impact the efforts of contributors before they submit resources to E3. This type of feedback will ensure that the project continually evolves to suit the needs of the community it serves.