This has been put together to address [https://bugs.eclipse.org/bugs/show_bug.cgi?id=214576 Bugzilla ER 214576].

−

+

−

+

−

+

−

+

== '''Terminologies/Acronyms''' ==

== '''Terminologies/Acronyms''' ==

Line 24:

Line 20:

== '''Purpose''' ==

== '''Purpose''' ==

−

We intent to set the quality expectations of COSMOS (the software) and matching acceptance criteria that would serve as a preamble to the COSMOS QA team while executing their work.

+

The purpose of this document is to set the quality expectations of COSMOS (the software) and matching acceptance criteria that would serve as a preamble to the COSMOS QA team while executing their work.

<p>

<p>

−

The QA team will use these criteria as a specification to define and plan all their testing efforts.

+

The QA team will use these criteria as a specification to define and plan their testing efforts for any given iteration. Specific test scenario's, environments and methods are to be documented in the QA iteration activities document eg) [http://wiki.eclipse.org/COSMOS_QA_i9_Activities COSMOS_QA_i9_Activities]

</p>

</p>

Line 33:

Line 29:

COSMOS QA perceives COSMOS 1.0 as being a well formed software if it is built upon valid use cases, on bug free technology, with accompanying API documentation and providing easy installations, on some widely used platforms.

COSMOS QA perceives COSMOS 1.0 as being a well formed software if it is built upon valid use cases, on bug free technology, with accompanying API documentation and providing easy installations, on some widely used platforms.

−

+

<p>

−

+

<hr>

{|{{BMTableStyle}}

{|{{BMTableStyle}}

|+{{BMTableCaptionStyle}}|Quality perspective 1: Is COSMOS well formed ?

|+{{BMTableCaptionStyle}}|Quality perspective 1: Is COSMOS well formed ?

Line 43:

Line 39:

|-

|-

| align="left" | Valid use cases

| align="left" | Valid use cases

−

| COSMOS team to define use cases [http://wiki.eclipse.org/COSMOS_Use_Cases COSMOS_Use_Cases]

| Owners take responsibility of the quality of content [https://bugs.eclipse.org/bugs/show_bug.cgi?id=197652 197652 (no target)]

+

−

| QA will not validate wiki content

+

|}

|}

== '''Is COSMOS 1.0 a consumable entity?''' ==

== '''Is COSMOS 1.0 a consumable entity?''' ==

−

COSMOS QA perceives COSMOS 1.0 as a consumable (adoptable) software if we can demostrate its capability to successfully integrate with participating data managers and MDRs through integration and performance testing reports, code samples and other adopter aids.

+

COSMOS QA perceives COSMOS 1.0 as a consumable (adoptable) software if we can demonstrate its capability to successfully integrate with participating data managers and MDRs through integration and performance testing reports, code samples and other adopter aids.

| Test the function and effectivity of helper tools – manual. Recorded as manual TPTP tests

−

|-

+

−

| align="left" | COSMOS stability during production deployments

+

−

| COSMOS team must state the minimum system requirements for production. Also recommend parameters (number of data Managers / MDRs that may be added, volume of data that can be queried, etc.) that should be considered for these tests.

| COSMOS team to define a process to integrate with the newer versions of these dependent software [https://bugs.eclipse.org/bugs/show_bug.cgi?id=215609 215609 (no target)]

+

| COSMOS team to define a minimum version of the dependent software [https://bugs.eclipse.org/bugs/show_bug.cgi?id=215609 215609 (no target)] [http://wiki.eclipse.org/COSMOS_M2_Dependencies COSMOS M2 Dependencies]

−

| QA will validate the process - manual

+

| QA will validate at minumum versions and sanity test at latest GA versions where this differs

|-

|-

| align="left" | Future enhancements / bug reporting mechanism

| align="left" | Future enhancements / bug reporting mechanism

Line 115:

Line 99:

== '''Operational Efficiency considerations''' ==

== '''Operational Efficiency considerations''' ==

+

+

The operational efficiency of a system is considered to be the characteristics of a that system when deployed to a production environment and it's reaction to load.

{|{{BMTableStyle}}

{|{{BMTableStyle}}

Line 121:

Line 107:

! Quality Characteristic

! Quality Characteristic

! Acceptance Criteria

! Acceptance Criteria

−

|-

+

! QA Role

−

| align="left" | Availability

+

−

| There are no Availability quality expectations

+

−

|-

+

−

| align="left" | Capacity

+

−

| There are no Capacity quality expectations

+

|-

|-

| align="left" | Concurrency

| align="left" | Concurrency

−

| How many queries / clients may run simultaneously? Should there be any other concurrency considerations?

+

| Execution of multiple processes or operations simultaneously. The COSMOS components are required to be multi threaded and support queries / clients to be run simultaneously. [https://bugs.eclipse.org/bugs/show_bug.cgi?id=216210 216210 (i9)]

+

| QA will perform basic concurrency testing by running 2 or more cosmos clients (command line and GUI) on same machine and by executing queries simultaneously.

+

|-

|-

−

| align="left" | Data Volumes

+

| align="left" | Data Volumes/Performance

−

| Are there any restrictions on the amount of data that may be returned by a query? How many queries / clients may run at a time?

+

| Restrictions on the amount of data that may be returned by a query/transaction. Data sets should be kept relatively small in a COSMOS implementation, however guidelines concerning data volumes and effect on performance should be determined and made available to adopters for a given scenario. More specific performance thresholds and metrics may be applied after further analysis of testing results. [https://bugs.eclipse.org/bugs/show_bug.cgi?id=216210 216210 (i9)]

+

| QA to perform generic load/data volume tests to generate data points for further analysis. QA will incrementally increase load on Example MDR repository, which is an XML file based repository and try to test sample queries to monitor response time, CPU and memory utilization.

|-

|-

−

| align="left" | Performance

+

| align="left" | Scalability/Stability

−

| The Data Managers and MDRs should not degrade the performance of Data Adapters by more than 15%

+

| COSMOS 1.0 will support a single Data Broker and Management Domain. The scalable quantity of Data Managers within a COSMOS environment will not be artificially limited and should not present a practical restriction in a well designed system, however, guidelines concerning the number of Data Managers and effect on performance should be determined and made available to adopters for a given scenario. Any COSMOS environment must be considered stable, with predictable results under all conditions. Exceptions should be handled gracefully with informative messaging to the client. [https://bugs.eclipse.org/bugs/show_bug.cgi?id=216210 216210 (i9)]

−

|-

+

| QA to perform generic load/data volume tests to generate data points for further analysis. QA will incrementally increase the number of MDR's, and subject to a constant transaction load of sample queries to monitor response time, CPU and memory utilization.

−

| align="left" | Scalability

+

−

| COSMOS 1.0 will support a single Data Manager and Management Domain. How many MDRs and Data Managers may be added? Should there be any other scalilbity considerations?

+

−

|-

+

−

| align="left" | Stability

+

−

| There are no Stability quality expectations

+

−

|-

+

−

| align="left" | Stress

+

−

| There are no Stress quality expectations

+

−

|-

+

−

| align="left" | Manageability

+

−

| System Administrator should have best practices, guidelines, and tooling to administer a COSMOS environment

+

|}

|}

Line 154:

Line 128:

This section includes the tasks required to complete this enhancement.

This section includes the tasks required to complete this enhancement.

# Shivvy, representing the QA team, is supposed to review and sign-off on these criteria

+

# SrinivasReddy Doma, representing the QA team, is supposed to review and sign-off on these criteria

== '''Open Issues/Questions '''==

== '''Open Issues/Questions '''==

Line 163:

Line 137:

A formal review is required to agree the content and detail of the quality expectations laid out in the three ‘quality perspective’ tables. This is a three step process…

A formal review is required to agree the content and detail of the quality expectations laid out in the three ‘quality perspective’ tables. This is a three step process…

−

1. Review each quality expectation to agree whether it should be included.

+

# Review each quality expectation to agree whether it should be included.

−

2. If it is to be included then determine whether the acceptance criteria can be met with an r1.0 timeframe, or be postponed to a subsequent release.

+

# If it is to be included then determine whether the acceptance criteria can be met with an r1.0 timeframe, or be postponed to a subsequent release.

−

3. All acceptance criteria must be linked to an ER that delivers the required quality. This can be illustrated by linking the acceptance criteria table cell content to either other relevant wiki pages or an ER. If the link is to an ER then this should also state the ER’s status (e.g. not started / WIP /completed)

+

# All acceptance criteria must be linked to an ER that delivers the required quality. This can be illustrated by linking the acceptance criteria table cell content to either other relevant wiki pages or an ER. If the link is to an ER then this should also state the ER’s status (e.g. not started / WIP /completed)

COSMOS Quality Expectations

Terminologies/Acronyms

The terminologies/acronyms below are commonly used throughout this document. The list below defines each term regarding how it is used in this document.

Term

Definition

Quality Expectations

Is a statement of some behaviour, characteristic or operational facility that a product must exhibit for it to be deemed ‘fit for purpose’. Quality expectations are normally grouped into four main categories: functional/behavioural, operational efficiency, inter operability factors; and admin/management factors (to control TCO).

Acceptance Criteria

This is a quantification of how a quality expectation is to be validated. For functional/behavioural quality expectations this is a simple Boolean test – it either works or it doesn’t. Hence, for most scope docs there is no need to specifically define functional acceptance criteria. However, other types of quality expectations – especially performance related areas – do require specific acceptance criteria because the quantification is normally some form of numeric threshold (with optional margin/tolerance) that states minimum levels of acceptable operational efficiency.

Purpose

The purpose of this document is to set the quality expectations of COSMOS (the software) and matching acceptance criteria that would serve as a preamble to the COSMOS QA team while executing their work.

The QA team will use these criteria as a specification to define and plan their testing efforts for any given iteration. Specific test scenario's, environments and methods are to be documented in the QA iteration activities document eg) COSMOS_QA_i9_Activities

Is COSMOS 1.0 well-formed software?

COSMOS QA perceives COSMOS 1.0 as being a well formed software if it is built upon valid use cases, on bug free technology, with accompanying API documentation and providing easy installations, on some widely used platforms.

Is COSMOS 1.0 a consumable entity?

COSMOS QA perceives COSMOS 1.0 as a consumable (adoptable) software if we can demonstrate its capability to successfully integrate with participating data managers and MDRs through integration and performance testing reports, code samples and other adopter aids.

Clear documentation, process definition for fixing COSMOS bugs or supporting future COSMOS enhancements can also contribute. Dependencies on 3rd Party software must be specified.

Operational Efficiency considerations

The operational efficiency of a system is considered to be the characteristics of a that system when deployed to a production environment and it's reaction to load.

Quality Perspective 3: COSMOS Operational Efficiency

Quality Characteristic

Acceptance Criteria

QA Role

Concurrency

Execution of multiple processes or operations simultaneously. The COSMOS components are required to be multi threaded and support queries / clients to be run simultaneously. 216210 (i9)

QA will perform basic concurrency testing by running 2 or more cosmos clients (command line and GUI) on same machine and by executing queries simultaneously.

Data Volumes/Performance

Restrictions on the amount of data that may be returned by a query/transaction. Data sets should be kept relatively small in a COSMOS implementation, however guidelines concerning data volumes and effect on performance should be determined and made available to adopters for a given scenario. More specific performance thresholds and metrics may be applied after further analysis of testing results. 216210 (i9)

QA to perform generic load/data volume tests to generate data points for further analysis. QA will incrementally increase load on Example MDR repository, which is an XML file based repository and try to test sample queries to monitor response time, CPU and memory utilization.

Scalability/Stability

COSMOS 1.0 will support a single Data Broker and Management Domain. The scalable quantity of Data Managers within a COSMOS environment will not be artificially limited and should not present a practical restriction in a well designed system, however, guidelines concerning the number of Data Managers and effect on performance should be determined and made available to adopters for a given scenario. Any COSMOS environment must be considered stable, with predictable results under all conditions. Exceptions should be handled gracefully with informative messaging to the client. 216210 (i9)

QA to perform generic load/data volume tests to generate data points for further analysis. QA will incrementally increase the number of MDR's, and subject to a constant transaction load of sample queries to monitor response time, CPU and memory utilization.

Task Breakdown

This section includes the tasks required to complete this enhancement.

Open Issues/Questions

A formal review is required to agree the content and detail of the quality expectations laid out in the three ‘quality perspective’ tables. This is a three step process…

Review each quality expectation to agree whether it should be included.

If it is to be included then determine whether the acceptance criteria can be met with an r1.0 timeframe, or be postponed to a subsequent release.

All acceptance criteria must be linked to an ER that delivers the required quality. This can be illustrated by linking the acceptance criteria table cell content to either other relevant wiki pages or an ER. If the link is to an ER then this should also state the ER’s status (e.g. not started / WIP /completed)