How To Test

This section lists IHE Profiles and actors whose testing is documented on this site. Each section documents tests, tools, and overall approach to testing. This site contains the reference list of tests and test descriptions that are loaded into Gazelle.

First a few Perspectives on XDS Testing

Server Testing

If you are testing a server, your server, then you probably have your software running on a development machine that is expecting to receive a network connection from the same machine or another local machine. Since you don't want to deal with firewalls (and corporate policies dealing with firewalls) it would be easier if a test client could be run locally on your development machine to exercise your server. That test client is called XDS Toolkit and is discussed more here.

Client Testing

If you are testing your client, a piece of software that expects to initiate a network connection, then you would rather test against a server on the Internet since you don't have to install and maintain the software representing the server. If the communications between your client and the test server on the Internet use standard ports and protocols then your corporate firewall is not going to get in the way of your testing, hopefully. The test server for XDS/XDR/XCA is called the Public Registry Server and will be discussed more below.

Other

There is actually a third category. If you implement a server that accepts a connection and then turns around and initiates a connection to another service. A good XDS example of this is the Document Repository actor which accepts connection requests from a Document Source and then initiates a connection to a Document Registry. This testing pattern is covered by a combination of the Server Testing and Client Testing procedures alluded to above.

Document Consumer (XDS.b)

All transactions for this actor are tested against the Public Registry Server. Generic WS endpoints are documented here. Individual tests may define specific required endpoints be used.

The Document Consumer actor is a client so it is tested against the Public Registry Server, specifically the Document Registry and Document Repository implementations.

All transactions are required to generate audit messages to the Audit Record Repository. Individual tests for this requirement are documented here.

Document Source (XDS.b)

The Document Source actor is a client so it is tested against the Public Registry Server, specifically the Document Repository implementation. Some of the tests defined below require that the Document Source actor be bound with a Document Consumer actor, allowing queries. Most of the Life Cycle option and Folder option tests are like this. A specific testing event may only require a subset of these tests to be performed.

Generic WS endpoints are documented here. Individual tests may define specific required endpoints be used. This allows for task-specific validations to be performed.

An example of a full Provide and Register.b transaction including the SOAP wrapper and HTTP message wrapper can be found here. An example of metadata necessary to submit a single document is found here. It includes a full description of the metadata elements included.

All transactions are required generate audit messages to the Audit Record Repository. Individual tests for this requirement are documented here.

Document Registry (XDS.b)

All transactions for this actor are tested using the downloaded XDSToolkit. Transaction specific tests are documented:

Configuring the Document Repository under test to forward to the Document Registry actor on the Public Registry Server

Using XDS Toolkit, submit a Provide and Register to the Repository under test, this results in a Register transaction to the Document Registry

Using XDS Toolkit, query the Document Registry to get the deposited metadata to evaluate it

Retrieves are tested by

Configuring the Document Repository under test to forward to the Document Registry actor on the Public Registry Server

Using XDS Toolkit, submit a Provide and Register to the Repository under test, this results in a Register transaction to the Document Registry

Using XDS Toolkit, query the Document Registry to get the deposited metadata

Using XDS Toolkit, retrieve and evaluate the document from the Document Repository

All transactions are required generate audit messages to the Audit Record Repository. Individual tests for this requirement are documented here.

Document Source (XDR)

The Document Source actor is a client so it is tested against the Public Registry Server, specifically the Document Recipient implementation. The Document Source actor in XDR is different from the Document Source actor in XDS in that it cannot be bound to a Document Consumer actor giving it the ability to perform Stored Query or Retrieve Document Set transactions. Thus, the following operations, normal for a Document Source in XDS cannot be performed by a Document Source in XDR:

Submit Document Replace/Append/Transform

Add to an existing Folder

The Document Recipient implementation on the Public Registry Server performs the following duties:

Validate that the submission is valid by the rules of XDR (above list)

Forward submission on to the Document Repository/Registry implementation for local storage (where more basic XDS validations are performed).

Because the XDR submissions are forwarded on to the XDS Repository and Registry, they can be:

Queried via Stored Query transaction and retrieved by Retrieve Document Set transaction even though this behavior is beyond the scope of XDR.

More importantly, these XDR submissions can be found in the Test Log just like XDS submissions which aids in diagnosing problems during testing.

The only transaction used in XDR is Provide and Register Document Set which requires MTOM/XOP web service formating for both request and response (example). Examples of metadata contents, contents of the RegistryObjectList element, are:

Initiating Gateway (XCA)

All transactions for this actor are tested against the Public Registry Server. Generic WS endpoints are documented here. Individual tests may define specific required endpoints be used. Transaction specific tests are documented:

Responding Gateway (XCA)

All transactions are required generate audit messages to the Audit Record Repository. Individual tests for this requirement are documented here.

Secure Node (ATNA)

The test collection for each of the above transactions includes requirements to test the transaction over a TLS protected connection as well as demonstrating the generation of appropriate audit messages.

Tool Overviews

For the testing of XDS/XDR/XCA there are basically two tools, the Public Registry Server and the XDS Toolkit. The Public Registry server is good for testing client software (software that starts by initiating a network connection). This server/service is available on the Internet year-round. The XDS Toolkit is a downloadable toolkit for testing your own servers (software that starts by accepting an incoming connection). This section gives an overview of these tools.

Public Registry Server

Internet site hosting reference implementations of:

Document Registry (XDS.b)

Document Repository (XDS.b)

Document Recipient (XDR)

Responding Gateway (XCA)

The Public Registry Server is itself the largest tool used in testing XDS/XDR/XCA. The key thing to know about this tool is the collection of Web Service Endpoints. They are documented here.

The rest of the tools, listed below have more technical documentation that can be found here.

Test Log

Database on the Public Register server that logs all events to the Document Registry, Document Repository, Document Recipient, and Responding Gateway implementations. The
Test Log Browser (discussed here) can be used to browse your entries to obtain detailed error status (sometimes beyond what is returned in error codes and error messages). The Test Log Browser is available here.

XDS Toolkit

The toolkit was originally a downloadable command line set of tools designed for Unix-like platforms. Some Windows users were successful at using it as well. This has evolved and now has a GUI interface and hosts a lot more tools than the original. The centerpiece of the old command line version was a command named xdstest which acted as a testing client initiating tests against a server under test. While the terms client and server are not used in IHE profiles, most understand the implications, a client initiates the network connection.

The xdstest tool was composed of several components:

Executable test engine

Test Repository (also called the test kit) which was a directory of test definitions interpreted by the test engine

Configuration files

Output area to hold resulting log files

The more modern form, the GUI based toolkit, incorporates these elements a wraps a GUI interface around them. In moving to the GUI form, several features were lost, at least temporarily. These will be re-incorporated in future releases. The key features that are still missing are:

Ability to run regression tests (such as run all Stored Query tests against a Document Registry)

Ability to send a Patient Identity Feed transaction to a Document Registry. Actually this was never part of the command line tool but will be available. This will send the HL7 V2 form of the feed. There are no plans to incorporate the HL7 V3 form

Documentation of a controlling API allowing integration into a continuous test/integration engine

One of the challenges of incorporating these features is that the GUI form of the tool is a multi-user tool. The command line form was not. The GUI is delivered as a Servlet (WAR file) to be run on a Servlet Container such as Tomcat.

Message Documentation

The primary documentation for IHE profiles is the Technical Framework. XDS/XDR/XCA/XDM are all documented in the ITI Technical Framework which is referenced by other IHE domains. While X* (XDS/XDR/XCA/XDM) references many standards, the OASIS ebXML Registry standard is the primary focus.

OASIS ebXML Registry defines two standards documents: ebRIM (Registry Information Model) which describes how to code various basic structures in XML and ebRS which defines a set of messages based on this information model. The information model gives us:

Name - short string label for an object

Description - longer string label for an object

Slot - name value pair (actually name and list of values)

Classification - classify or label an object, especially to support search. Medical codes are built on this foundation

ExternalIdentifiers - externally assigned identifiers intended to be used as primary indexes. Unique ID and Patient ID are examples

These basic types are modeled within X* to describe DocumentEntries (description in the registry of a real document in a repository), Folders (organization of DocumentEntries), and SubmissionSets (for documenting the submission (who/why/when/...). The documentation is published in Volume 3 of the ITI Technical Framework. It lives separately since it is referenced by all the X* profiles. The web service standards referenced by X* (and many other IHE profiles) are documented in ITI Technical Framework Appendix V.

Besides the Technical Framework, the ITI Technical Committee maintains an Implementation Guide, a set of wiki pages with additional guidance to developers. It is found here. The primary resource for testing is here which acts as a cover sheet for the page you are now reading and many others.

Technical Resources

The validation of individual messages relies on several tools based on industry standards.

Schema

The following XML Schema files are used to perform basic XML message validation:

From the OASIS committee:

ebRIM - information model

ebRS - registry services (message descriptions)

lcm - lifecycle management requests

query

ITI publishes an additional schema that wraps the above along with modern web services to create the Provide and Register.b and Retrieve Document Set transactions:

XDS.b_DocumentRepository

Metadata Validator

One cannot live by schema alone. XML Schema cannot describe every nuance of an XML message. The data structures defined in ebRIM are somewhat complex. Given the modeling IHE has done on top of ebRIM to create the XDS Metadata Model, XML Schema is not adequate to describe and validate the DocSharing messages.

The Metadata Validator was created to evaluate ebRIM based messages against the XDS Metadata Model. It is available as part of the XDS Toolkit (online version http://ihexds.nist.gov:80/xdstools2 under the menu item Message Validator.

Affinity Domain Configuration

Also known as codes.xml.

An XDS environment is based on an Affinity Domain which includes an agreement on the part of the participants as to the specific codes that can be used to encode various metadata descriptions of documents. Coding agreements are a huge part of interoperability. XDS does not mandate the use of a specific set of codes. Instead it mandates that the collection of systems bound into an agreement to form an Affinity Domain document a set of codes to be used. The idea is that an XDS actor needs to be configurable to adapt to the local configuration.

For the Public Registry, the Affinity Domain configuration is encoded as an XML file named codes.xml. We keep this file up to date by accepting requests from testing organizations and individual developers to add codes to this configuration. All messages sent to the Public Registry are validated against this configuration, part of the Metadata Validator tool. The current version of codes.xml can be viewed here.

The following specifics are documented in codes.xml. Note, there is no formal documentation on exactly what must be part of an Affinity Domain configuration. We have continued to update codes.xml as a living example of the best ideas that have come forth.

Metadata attributes based on the Classification XML element are health care codes (code value, code name, code definition scheme tuple). SubmissionSet.author and DocumentEntry.author use Classification but are not codes. Examples are: DocumentEntry.classCode and DocumentEntry.confidentialityCode and DocumentEntry.formatCode

Mime Types that are acceptable for submission

Patient ID Assigning Authoriy

WSDL

WSDL (Web Services Definition Language) descriptions of web services has become an industry standard approach to describing how XML Schema is mapped to individual messages that are part of a service. WSDL describes a set of services, the messages that make up the requests and responses to those services, and the XML Schema descriptions of the messages. Tools can be used to generate code stubs in various languages to parse and generate messages based on WSDL descriptions. IHE publishes Informative WSDL for all its web service based profiles.

Normally WSDL can be retrieved from an endpoint by appending ?wsdl to it. Unfortunately the technology used in the Public Registry server does not support this feature. It will return WSDL. But it will not contain the correct information.

Implementation Notes

Where to ask questions

We have a mailing list dedicated to XDS/XDR/XCA (and related profile) implementation and testing. It is ihe-xds-implementors@googlegroups.com. Discussion of ITI Profiles belongs on the ITI committee mailing list ititech@googlegroups.com. If you need to contact me directly, I have a new email address for IHE work bmajur AT gmail dot com.

Testing

Test Documentation

a table of all the XDS/XDR/XCA tests indexed by the XDS transaction they support. Includes links to the documentation for each test. This link always points to the current year/season. This table is used to load Gazelle/Kudu so if you find discrepancies please let me know.

detailed description of all the tests with instructions for executing. You can look directly here if you know a test number. If you want to look up a test by actor or profile use the link in the previous section.

All XML Schema files used to validate Registry/Repository related transactions.

Rules for submitting results to Gazelle

In the past vendors were expected to upload test logs into Gazelle for validation. This is not longer being done. The tooling will tell you whether you passed or failed a particular test. If you don't do a complete job of testing of your systems during Pre-Connectathon then Connectathon will a very painful experience. Most developers only make this mistake once.

Testing Tools

This is the tool for testing your server actors (actors that operate by accepting a transaction to start their work. Examples are Document Registry and Responding Gateway). It can be downloaded for local use and there is a copy installed on our Internet server. The toolkit comes pre-configured with the endpoints for the Public Registry so it can be used to submit/query/retrieve to the Public Registry. This is sometimes valuable for debugging your own work.

A testing oriented implementation of XDS Document Repository and Document Registry as well as a limited Responding Gateway. This service is available on the Internet and on the local network at the North American and European Connectathons. The endpoints for the Internet copy are available here.

a web page with a collection of tools that operate on the Public Registry Server. It includes the following tools:

Patient ID allocator for the Public Registry - this Document Registry service implements the restriction that all Patient IDs must be registered before submitting metadata on their behalf. This service allocates a new Patient ID and registers it.

a web based tool for browsing the test logs maintained on the Public Server. It detects the IP address from your browser connection and only shows log entries for transactions sent from that IP address.

Public Registry Server Configuration

The Public Registry is a server maintained on the Internet and is generally always available for private or IHE sponsored testing.

It resides at ihexds.nist.gov (129.6.24.109).

Configuration

that defines the Public Registry Affinity Domain configuration. This includes definitions of acceptable codes, mime types etc. The Metadata Validator tool (web page) is available for you to manually test your metadata. The same validator is used to check every metadata submission to the Public Registry (ProvideAndRegisterDocumentSet and RegisterDocumentSet transactions). Specifically, it passes the submitted metadata through Schema validation and then metadata validation. Metadata validation has two parts: validate that the metadata is structured according to the rules of XDS (which are more strict than ebXML Registry standard) and that the codes used in the metadata are consistent with the configuration of the Affinity Domain. Updates to this table are made on request to support community activities.

Known bugs in the Public Registry

The Folder metadata object maintains a Slot named lastUpdateTime which reflects the last addition to the Folder. If this Slot is present on submission it is overwritten with the current time. But, currently if this Slot is present in the submission a second Slot with the name lastUpdateTime is added to the Folder resulting in a Duplicate Slot error from the backend registry.

The Responding Gateway is not a true gateway but instead a simple interface to the Registry and Repository. It's functionality is adequate to test for management of the homeCommunityId attribute but little else.

Test Event Configuration

The proper endpoint to use for testing is determined by the following:

The service (Registry vs. Repository vs. a specialty service designated for a specific test)

Which testing event you are preparing for. Each event issues its own collection of certificates for TLS so using the proper endpoint will give you access using your issued certificate.

In the tables below, these issues are reduced to three 'variables':

EVENT

the non-TLS port number to use

SEVENT

the TLS (secure) port number to use

Important URLs / WS Endpoints

Event Configurations

Event Name

EVENT

SEVENT

Available

North American Connectathon in Cleveland in January 2015

12090

12091

Yes

European Connectathon in Vienna in April 2014 - Uses same certs as last year (EU 2013)

12090

12092

No

NOTE: There are several TLS configurations available. See here for details.

Combination of xcaregistry and xcarepository - accepts Stored Query and Retrieve. This endpoint accepts XDS and not XCA transactions. See test Test 12342 for details.

Stored Query

http://ihexds.nist.gov:EVENT/tf6/services/xcaregistry

https://ihexds.nist.gov:SEVENT/tf6/services/xcaregistry

Retrieve Document Set

http://ihexds.nist.gov:EVENT/tf6/services/xcarepository

https://ihexds.nist.gov:SEVENT/tf6/services/xcarepository

Asynchronous Web Services

Provide and Register - b

http://ihexds.nist.gov:EVENT/tf6/services/repositorybas

https://ihexds.nist.gov:SEVENT/tf6/services/repositorybas

Register - b

http://ihexds.nist.gov:EVENT/tf6/services/registrybas

https://ihexds.nist.gov:SEVENT/tf6/services/registrybas

Stored Query

http://ihexds.nist.gov:EVENT/tf6/services/registrybas

https://ihexds.nist.gov:SEVENT/tf6/services/registrybas

Retrieve Document Set

http://ihexds.nist.gov:EVENT/tf6/services/repositorybas

https://ihexds.nist.gov:SEVENT/tf6/services/repositorybas

Cross Gateway Query

http://ihexds.nist.gov:EVENT/tf6/services/rgas

https://ihexds.nist.gov:SEVENT/tf6/services/rgas

Cross Gateway Retrieve

http://ihexds.nist.gov:EVENT/tf6/services/rgas

https://ihexds.nist.gov:SEVENT/tf6/services/rgas

Many tests use specialty URLs representing special testing services. These are documented in the individual tests.

This server is configured to accept/return the following configured values

repositoryUniqueId is 1.19.6.24.109.42.1.5

homeCommunityId is urn:oid:1.19.6.24.109.42.1.3

Specialty Error Endpoints

These endpoints have been added to the Responding Gateway to help test handling. These endpoints always generate the specified error, independent of inputs and most conditions. They should be used to verify your handling of each kind of error. All errors allowed by the XCA profile are included. Notice that each endpoint (last section) starts with either rgreg or rgrep. Each endpoint accepts either a Stored Query or a Retrieve Document Set transaction. With this separation it should be possible to test an Initiating Gateway just by manipulating the configuration.

Specialty error endpoints for Responding Gateway

Error generated

Transaction

endpoint

XDSMissingHomeCommunity

Query

http://ihexds.nist.gov:EVENT/YEAR/services/rgregmissinghome

XDSMissingHomeCommunity

Retrieve

http://ihexds.nist.gov:EVENT/YEAR/services/rgrepmissinghome

XDSUnavailableCommunity

Query

http://ihexds.nist.gov:EVENT/YEAR/services/rgregunavailable

XDSUnavailableCommunity

Retrieve

http://ihexds.nist.gov:EVENT/YEAR/services/rgrepunavailable

XDSUnknownCommunity

Query

http://ihexds.nist.gov:EVENT/YEAR/services/rgregbadcommunity

XDSUnknownCommunity

Retrieve

http://ihexds.nist.gov:EVENT/YEAR/services/rgrepbadcommunity

TLS

The Public Registry Server and the XDSToolkit support only two cryptographic cyphers for TLS:

We have not yet found a system that did not have one of these available in their default set (if necessary we will add to this list in the future)

North American and European Connectathons typically utilize different style certificates. North America usually issues self-signed certs while Europe usually employs a certificate authority. Usually the issued certificates are valid for a period of one year (until next year's testing season). A TLS-enabled port is available for each certificate set.

Updated certificates are loaded onto the Public Registry Server each season. Likewise, a new version of the XDS Toolkit is issued each season containing the new certificates.

Syslog Facility

This service has been removed

The Public Registry server has a syslog facility to be used with joint XDS/ATNA testing.

A browser (here) is available to view your audit messages. This browser will only display your audit messages to you. It does this by sensing your IP address and only displaying entries generated from that IP.

The browser includes a validator that evaluates the content of a syslog message against the requirements published in the Technical Framework. To validate one of your messages:

Find your message in the browser and select it with the mouse

Details of your message will appear on the right side of the screen

Select the Validate link on the top of the right side

The validation results will be displayed in a separate browser tab/window

NOTE: The validator is limited to the following transactions. Other transactions, when validated, will give a large list of meaningless errors. The validator currently understands the audit log requirements of the following transactions:

ITI-14

ITI-15

ITI-17

ITI-18

ITI-41

ITI-42

ITI-43

A mechanism for you to get credit for your audit message during Pre-Connectathon testing (also called MESA testing) will be published soon.

Test Data

Understanding EVS

This section is obsolete. We no longer use EVS nor do test grading.

XDS and related tests require the vendor to upload evidence files to Kudu/Gazelle for evaluation. Successful evaluation results in tests being marked as passed. This section describes the process of evaluation and how to use it.

Tests that can be auto-evaluated have a button NIST Validation on the Connectathon Manager's Mesa Log Return page. It is found in the Logs column. Pushing this button does:

The uploaded log files are bundled into an EVS request to the NIST Public Registry Server.

The result of this request is displayed as XML at the top of the page.

If the result is Success then the relevant test is marked Ok.

If the result is Failure then the ugly XML displayed must be scanned for the first <error> element. The text content of this element is the error message.

For Connectathon Managers, this error text should be evaluated to see if it is a tooling problem or a vendor testing problem:

If the error text looks like Validating test 11995, which contains section eval but no log for that test/section was submitted, found [11994/rplc, 11994/xfrm, 11994/submit, 11994/eval] instead. then the problem belongs to the vendor.

Other typical vendor mistakes are wrong endpoint used on the Public Registry server and results include tests that failed

Most other error messages are really tooling problems

Vendor error messages should be copied into the Comment box and the test marked as Failed

Tooling problem messages should come back to me to investigate

There is one class of tooling problem messages that should go back to the vendors. The Java platform I use has a well known bug decoding some ZIP files. This occurs if the ZIP compression uses a particular type of coding. The error indicates that the tool cannot decode the ZIP file. This seems to occur in one out of twenty ZIP files. In this case all that can be done is for the vendor to remove the ZIP file from the upload area and in its place upload the individual log.xml files. They will need unique names (within their set of files) but the file names are not important to validation. When the Java platform posts an update that fixes this bug this error will no longer appear. Error messages that are shown because of this bug are:

EvsWrapper#getLogsByTest: found 2 Test elements in log, the testlog file must be corrupted

WS:Security

For select projects, the Public Registry server and XDS Toolkit support WS:Security headers and the inclusion of SAML Assertions (XUA profile).

Public Registry

WS:Security is enabled on the server side by making a separate instance of Axis2 available with Rampart enabled. This means that all connections to these endpoints must include a WS:Security header or the connection will fail. These endpoints are only offered with TLS enabled. The minimal WS:Security header includes a timestamp and the signature of the timestamp. Rampart will verify that the header exists, that it has a timestamp, that the timestamp is signed and that the signature can be verified.

Separately, if a ws:Security header is included, it may contain one or more SAML Assertions. No validation is done on the contents of the SAML assertions at this time.

XDS Toolkit

The toolkit can generate ws:Security headers with or without SAML Assertions. The following conditions are necessary to successfully generate ws:Security headers in toolkit generated messages:

actors.xml must be configured to point at a site ready to accept ws:Security based connections. As delivered it includes a site nhin which points to the ws:Security enabled endpoints on the Public Registry server

Certificates must be installed in two places:

xdstoolkit/xdstest/keystores/EVENT/keystore - where EVENT is selected in the xdstest script as XDSEVENT. This certificate governs authentication and encryption of the TLS connection.

xdstoolkit/xdstest/rampart/client_classes/client.jks - this certificate is used to sign headers for ws:Security. The client.properties file (same directory) describes the certificate.

xdstest must be run with the --wssec command line flag (to engage Rampart/ws:security) and the --secure (or -S) flag to engage TLS

The toolkit can send SAML Assertions as part of the ws:Security header. If the --wssec command line option is specified and if the testplan has a <WSSECHeader/> in the transaction setting then the assertion contents of the <WSSECHEADER/> will be inserted into the ws:security header, signed and sent. It is expected that <WSSECHeader/> will have a single child, <Assertion/>. Here is an example, empty, assertion shown in the context of a StoredQueryTransaction tag in a testplan:

Open Source

XDS/FHIR Metadata Mapping

Recent Updates

17 May 2010 - posted copy of XDS GUI Toolkit that was used at Euro2010 Connectathon. Brief installation documentation and pointer to download is available here. The configuration editor has been fixed.

17 Feb 2010 - The certificates have been distributed for pre-Connectathon testing for Bordeaux. The Public Registry server will accept these certificates on port 9443 (see here for details).