The purpose of this document is to enumeration the known
requirements against the Hyades project in the next few quarters.

The intention is to provide Hyades GA level drivers quarterly,
and this document will help communicate what the pressures are and how they are
being prioritized. This document should not be taken as a commitment of all the
things being developed. It should also not be assumed to be a complete or fixed
list of requirements. The project is dynamic in many senses including the
staffing and the requirement pressures.

Throughout the
document you will see a notation like “T23(P1 1.3)” or “(P2 1.2)”. This is a
reference to the theme the requirement is linked to “T23”, the priority
assigned “(P2” and the target release “1.3)” for that priority. The objective
is to describe the context of the requirement and provide a proper amount of
support for each theme. Requirements with only a theme identified are not under
consideration for the 1.1, 1.2 or 1.3 time frame.

Briefly, P1 means
the release can not ship without the function, P2 means the function is highly
desirable for the release and all effort will be made to do it, but it is not a
stop ship function. P3 is basically a nice to have. Note the priority is not a
staffing commitment by the core Hyades development team, and we are always
looking for help and contribution.

The releases are
intended to ship quarterly. 1.0 was targeted for the end of June 2003. 1.1 is
targeted for the end of September 2003. 1.2 is targeted for December 2003, and
1.3 is March 2004.

At this time (summer 2003) the closer of both these
standards is still underway. Until both of these standards are finalized Hyades
will simply monitor them and feed back as appropriate. A commitment can not be
made until closure is had on these standards, and in particular it is
determined how best to get access to a UML2 model in the Eclipse environment.
There are several discussions underway for projects in the UML2 area, but until
something concrete and public is available Hyades will have to continue on its
current path of building a wrapper service to abstract the needed structures,
and maintain a façade over this area.

It often occurs that both a user and a tool developer need
to modify code that has already been deployed. This could be for the purpose of
adding a “System.out”, a logging event, or instrumentation to for the purpose
of code coverage analysis among many other uses.

Hyades will provide a framework for doing this type of task,
as well as a few tools that leverage this capability.

The base toolkit will provide a runtime component that can
be used (at class load time for Java) to modify the byte stream of a class.
There will also be an editor for creating or modifying the definitional
“script” for a given probe.

For convenience of the probe writer there will also be a set
of libraries available for the common tasks of logging and tracing that
integrate well with the rest of Hyades.

In Hyades 1.0.2 a HTTP recorder and playback sample tool was
provided to demonstrate how the Hyades model and data collection infrastructure
could be used for this task. This was basically a generated URL blaster. In
stages over 1.1 and 1.2 a more robust HTTP stress testing tool will be
provided.

Support for SSL recording, data pool exploitation, and mixed
user loads will all be provided.

In combination to the other data collection support in
Hyades, this can be a very powerful tool when you are stressing and fixing your
application.

Tools that generate code into Eclipse projects should set
cross project dependencies automatically to allow for a simple build process.
This means a close working resolution of this issue with the team support
environment provided in Eclipse. In addition test projects should be able to
refer to each other just as Java projects can, and the equivalent of build
dependencies when dealing with the namespaces used in code generation for
example.

A common use case of tests is to have them reused in
slightly different execution environments. One thing that is required for this
environment is that configurations need to be customizable for a specific
deployment or run of the test. Not only does this imply that deployment time
modifications need to be possible, it should be considered if for audit reasons
those edits should be saved. In the audit scenario we recommend the change
management system should be used and versions of the configuration saved.
However temporary edits should be allowed by the Hyades user interfaces.

A feature that was targeted for 1.0.0 and did not make it
was support for model loaders to be able to have their own extension points.
The prime purpose for doing this is to provide a means for multiple handlers
for a given fragment arrival. The current structure matches exactly no or one
loader for each fragment type, however often there may be a need for further
awareness of the event arrival. This may be before or after the data has been
reflected in the model. A general notification scheme could be used, but it is
felt that this would be too heavy a solution for this problem. Basically when a
fragment arrives, loaders and other handlers need to be invoked in arbitrary
order. Once a loader is about to run, or finished running another set of
handlers need to be enabled. This should not require a resource save or any
other state condition. Just the time situation of preload, load and post load
time need to be trigger points. An example usage may be that as data arrives
further stimulus needs to be sent to the execution environment. An automated
tuning use case would be an example. A responsive stress testing tool would be
another.

Another feature needed in the near term for Hyades is the
ability to more dynamically control a test environment. The user needs to not only
be able to run a test, but they also need to stop, pause and resume it. In
addition batch executions need to be able to be dynamically attached to for
monitor and control.

Performance analysis, test drivers and other tool scenarios
supported by Hyades sometimes require the capturing of parameter values when
recording or tracing. These may be needed for analysis or simply used in the
playback of a recording.

A natural exploitation of being able to capture values
during a trace is to use values during an execution. Although the support of
capturing values and using of values can be separated, they have so much in
common at the infrastructure level we will link these two capabilities
together.

An important facet of using values at execution time is the
notion of a data pool. Hyades needs to support set of data pools at execution
time to allow for variable substitution of static and live data. This implies
both a data storage facility for large volumes of static data, and runtime
access to random data generators and other possible sources.

There is a requirement to allow editing environments running
outside Eclipse to be able to interact with a Hyades project.

Although importing and exporting projects is naturally
supported, there are no current plans to provide addition external tool
integration beyond the basic plugin and resource integration provided by the
base Eclipse workbench and the Hyades components themselves.

The general objective of Hyades is to handle more than Java.
For example a more direct support for protocols like HTTP/S is needed to best
exploit that type of record and playback. In addition languages such as C# are
important players in a distributed environment.

Beyond the basic definitional recognition that these other
languages needs to be supported, the execution environment needs to provide a
native binding for these languages without any unneeded heavy cross language
infrastructure. That said, fortunately the data control and communications
layer of Hyades is native code with additional Java bindings. Some additional
work is needed to provide equal work at the native level and this is the near
term requirement on the execution environments that wish to directly drive with
these non-Java language environments.

As we have grown the content and capability of the Hyades
tools, the use of the profiling and test perspectives need to be re-visited.
For example, what is the appropriate place to integrate code coverage views? In
addition, linking performance profiling to debug views may be a natural way to
integrate with the JDT for example.

Team support needs to be improved. Cross project references are
facets of complexity in this problem area, however cross artifact associations
are the problem area. This is amplified by the lack of artifact comparing
tools. With structured resources using EMF we have a concise representation of
the information, but this representation is tool friendly versus user friendly.
Not only do we need to have these comparison tools, these tools need to deal
with the implication of differences at the other end of an association.

For example the class path may be captured in the configuration,
but if the test structure or SUT changes at a packaging level, the class path
may need to be changed. This is the first step of doing impact analysis. Full
impact analysis does not need to be support in the first pass, but it needs to
be facilitated by the models and the user interfaces and extension points.

Another important concept that needs to be supported and
exploited in Hyades is the notion of typed agents. This means that agent types
and names are both captured in the model as well used to enable various user
interface metaphors such as filter settings and agent selection. This implies
more selective control in the user interface of the agents being used, plus
visibility to the user what he agent actually does. The control is already possible;
however it is not exploited by the UI or the agents.

Continued focus on speed and footprint will be required as
well. The overall memory needs of the Hyades components needs to be continually
reduced. Importantly at this time the tools and EMF itself work on a memory
model of everything is live. This is not acceptable for tools that deal with
large volumes of data, yet the problems in this area need to be hidden from the
end user. The intention in Hyades is to make this scaling possible and allow
the higher volume solutions become a value add vendors can plug in to the base
Hyades tooling and services.

Finally there are always new realizations in the area of
usability. A constant effort to continue improving in the area is key to
project success. We have several requirements for increase ease of use and
scalability for the very basic Log Analysis viewer and as with all requirements
in this area, there will be incremental progress in each release.

Hyades already supports profiling data collection from the
command line for cases where launch and/or attach from a workbench is not
appropriate. The test execution and collection environment needs to provide
same support. In other words, a user should be able to run a test suite and
have all events log to a single or small number of files that can later be
imported into the workbench.

Naturally the report generation capabilities in the test
environment also need a “GUI less” way of being invoked as well. This will
better enable large scale projects and well as easier reuse of the assets
Hyades can create.

One typical quality metric associated with testing is
coverage. This is sometimes practically limited to function level coverage, but
is normally thought of as a line level or logic block level metric. This is a
natural extension of the trace support provided in Hyades, but adds unique
visualization and use cases to the data.

Hyades will need to provide complete infrastructure to
handle all levels of coverage collection and reporting, but will need to
provide at least a function/method level coverage tool as example function.

Execution engines need to be able to run other execution
environments. In its simples form, a simple service to invoke Java
implementations from other Java implementations would be okay. However this
will lead to a discussion that will end up defining an important part of a
generic testability interface.

Hyades definitely needs to work on defining both a generic interface
to execution environments as well as to the testable interface of a SUT.
Initially getting closure on the execution environment is appropriate and work
to resolve the rest may be longer term.

Hyades provides a log monitor/import/analysis capability.
Many tool providers will add their own log imports; however the current
mechanism suggests a static parser be written. Many log formats are very
predictable and can be parsed using basic regular expressions. A basic generic
adaptor that is configurable for various pluggable data sensors, parsers and
formatters will be provided in Hyades the ease the effort of adding more data
sources.

In addition to recognizing basic log entries, a commonly
logged piece of data is a stack dump and this can be more optimally captured
than the current basic CBE.

Hyades provides a statistical model for storing numerical
data over time. The intent was for this model to be used to store such things as
system resources for viewing and analysis during load testing for example.

Now that we have a model, we need a data collection agent
and a generic viewer to show trends graphically and a table view for the raw
data. It has been suggested that we do a “perfmon” style data collector first
as a sample data collector for this type of collection.

Hyades has developed a set of models for some domain
specific usages. However the roots of Hyades are the linkage of these domains.
It has already become apparent that we need to accelerate the linkage at the
model level. For example we know intuitively there is at least a logical
linkage between the behavior of a test case with the interactions captured in a
trace, and we are working on this link in sync with the evolution of UML2.

Another important usage of the kinds of data collected in
Hyades it the possible association of data in a log with data in a trace or a
particular test execution. Another example is the capturing of statistical data
during the run of a test or trace scenario.

These associations need to be made available in the Hyades
models, but in a way that does not increase the cost of the models.

Often data collection can be done live and incrementally at
execution time.

In fact this style of collection model has been avoided
historically in order to deal with data compression immediately. Particularly
in performance collection, the notion of collecting trace styled data has been
avoided and the type of analysis desired was used to define compression
techniques at collection time. Hyades has promoted trace style collection for
example in order to simplify the collection agent. In effect this means the
collection agent stays simple and sends trace data when ever it is collecting,
and the data is filtered by the agent and the listening workbench where it is
used to populate the model for various types of analysis, potentially avoiding
re-runs of a scenario.

However because there are existing data collectors that
locally aggregate, and there are data volume and disconnected scenarios that
require compression of the data, Hyades needs to allow this data to be loaded
without having to simulate tracing. The models are already capable of handling
this, but the fragments that need to be loaded are not defined nor are the
loaders provided.

One of the more difficult things to deal with in a large
application is often the understanding how things actually flow once all put
together. This is a critical piece of information for basic maintenance as well
as more detailed performance tuning work.

Ensuring that each module
developers write is correct, robust, and maintainable is an integral part of
the process one should follow to deliver quality software. Static analysis
techniques are commonly used to automate this process of reviewing code and
verify that it complies with a set of rules. Rules typically range from readability
to performance improvement, to best practices enforcement, to defect detection.

The goal of this framework is to
provide a set of extension points for rules engines to plug onto Hyades. The
purpose of these extension points is to provide a means to report results of
such analysis as a test phase result. Additionally, the framework will provide
a generic rule engine along with a rule schema and a basic set of rules
definitions.

In addition to new scenario support, which will drive
changes to the base framework there are enhancements to improve the existing
user experience. For example once paging lists are leveraged in the models, the
UI needs to be reworked to exploit this memory footprint optimization (P2 1.3).

The interface needs to exploit agent types and names to give
the user more control.

The following are broken down to the various topical areas
of the user interface code.

Test

T3 (P1
1.1) New HTTP Recording interface and test editor/composer.

T2 (P1
1.1) Provide a probe script editor and other UI for the probe toolkit

T5 (P1
1.2) Provide editor support for data pools

T9 (P2
1.2) Add the capability to run a test suite from the command line and
collect the events into a log file for later import.

T9 (P2
1.2) Add the capability to launch a test suite but collect the data in
standalone mode. In effect this is a launch and don’t monitor scenario

In general there is a need to provide a standard “best
practices” way for vendors to extend the models without requiring the inclusion
of their changes to be in the open source project, yet still allow data
interoperability with other select vendors.

(T8) Models also need to have comparison tooling, and this
may require some common services at the model level to make merge easier to
manage.

(T4 (P2 1.1))Model loaders need to have a more flexible
framework to execute in. Namely handlers need to be able to be registered
dynamically for fragment arrival (pre-loader), load time (loaders), load complete.
This should not be tied to resource state as there is already sufficient
mechanisms in place on resource state change boundaries.

(T8) Continued performance tuning in the areas of footprint
and speed, as well as low cost dynamic model extensibility is always important
to continue focusing. A specific area of improvement will be to leverage the
“paging list” support being developed in EMF which will allow unreferenced
objects in list to be garbage collected (P2 1.2). Assuming paging lists are
completed and exploited, making the addition step to optionally back the models
in a relational database versus XMI file would be the next level of scalability
(P1 1.3)

T10
(P2 1.1) Additional annotations are needed for complete coverage data
collection. This may involve a link to the statistical model for more
optimal storage of the data.

T10
(P2 1.1) Extend TRCAggregatedMethodInvocation to carry baseTime,
cumulativeTime, minTime, maxTime, and callSite; collected for a given
invocation along a unique call chain

T15
(P2 1.1) Add explicit support for pre-aggregated data, which implies new
loaders and fragment definitions

T14
(P1 1.2) Add a link to associate a log or a log entry to particular events
in a trace model.

T10
(P2 1.2) Support line level accumulators in TRCMethod or TRCSourceInfo

T7 C#
support

T7 C/C++
support

T8
There is a need as part of the footprint reduction effort to support more
compact persistent storage formats than the current zipped XMI files.
Binary formats are often discussed in this are, but it needs to be
carefully balanced with the CPU costs of transformation.

T7
(P1 1.1) Cross language linkage to the execution environment which implies
any code generation would be to these native languages.

T8
(P1 1.1) Standalone execution capabilities which require no workbench
interaction and produce appropriate event logs. The execution environment
“agent” by definition can be attached to when it is running, however, the
various combinations of behavior possible need to be worked out. For
example when running standalone, and then the agent becomes monitored,
should all the existing history be flushed to the workbench?

T4
(P2 1.1) Support a more interactive control model that allows execution to
not only be started, but also paused, restarted, stopped and cancelled.
This will include events to reflect these execution state changes.

Have a way to display (unknown) nodes and associated
actions in the Test Navigator.

Indeed, the Hyades model is candidate to be extended by a
tool vendor. For example, the SUT object is generic and could be re-defined to
a link to the source code under test and verifications on this test.

In that case, the test case would have two different
children:

- a link on the source code

- a test verification

The associated tree could be something as:

Test Suite

|

|--Test case 1

||

||--verifications

|||

|||--assert1

|||--assert2

|||--

||--SUT

|||

|||--sourceFile1.java

|||--sourceFile2.java

|||--sourceFile3.java

|||

|--Test case 2

|

|

|--Test case 3

|

|

A right click on a "verification" node could
display a sub menu "test report".

So if I summarize:

- to have a way to display new nodes in the test
navigator,

- to have a way to define action on these nodes.

First, I thought about a solution based on extension
point, but if it exists something simpler it will be fine.