The goal of the SSOCheck API is to develop own testing clients and logic or even use a standard browser to execute SAML 2.0 Browser Profile tests. You can control, watch and monitor the behavior of the test client in any step of the test sequence. This is a main differentiator to tests that are only run server side.
The following diagram outlines how a test run looks like and how the SSOCheck API should be used.
You can easily see that the sequence is following a normal SAML 2.0 Browser SSO POST Profile except the steps colored in green which call the API and translate the standard SAML response into a modified SAML test response which can then send to the service provider. The response of the service provider needs to be evaluated and interpreted in order to get a test result.
Please note: In case of internal mode usage the “normal” SAML response returned from the IDP is an encoded response that needs to be translated by the API to get the modified SAML test message.

Can I use SSOCheck only in combination with a SSOCircle hosted IDP? No, you can use it with your own IDP. SSOCheck API offers two different modes of operation. The API can be used in an “external” and an “internal” mode.

External Mode
External mode means that you can use any identity provider and you are going to send the SAML request generated by the external IDP to the SSOCheck API which in turn generates modified SAML messages which then can be sent to the service provider. As messages transporting assertions are signed and cannot be modified without tampering the signature, “external” mode is currently limited to signature related tests.

Internal Mode
Internal mode is using a SSOCircle hosted service as the Identity Provider (either the public IDP idp.ssocircle.com or the white label hosted IDP idpee.com). Internal mode requires that a trust setting must exist between the service provider tested and the identity provider at SSOCircle. A prerequisite for that is to configure the hosted IDP and exchange meta data.

Internal mode has the advantage that SAML message are created from the ground up and can be modified at any stage and at the end sign the message. In internal mode more test options are available. The exact number of test depends on the SAML configuration.
In internal mode a test client has to act as a browser (following SAML profiles) and to query the SSOCheck API (to get test messages and test meta information). On acting as a browser the client is required to send specific headers to the IDP. The headers allow the IDP to recognize the request as a test request and keep state in a specific test plan run.

Test types and results
What type of tests are executed? A test plan should always start with a positive test: running a SAML SSO without modifying the SAML message. The test should always return with a “success” result. It hat is not the case, continuing with the test makes no sense.

We are continuously modifying and adding tests. The current valid list of test can be seen from the test plan configuration page. A coarse list of test categories are listed below:

Conformity tests of a Service Provider against the SAML standard

Security tests

A short and not complete overview of individual tests:

Protocol test / Message replay test

Modification of the SAML response message

Changing timestamps

Changing status code

Changing Issuer

Changing versions numbers

Modification of the SAML Assertion

Changing timestamps

Changing status code

Changing Issuer

Changing SubjectConfirmation data

Changing audiences

Changing valid time windows

Changing SAML conditions

Changing NameID

Changing version

Changing AuthNStatements

Signature tests

Signature exclusion

Tamper the signature

Sign with an invalid key

signature wrapping

About test results: success, fail and warn
The expected and correct result of a test, a success may be a error 500 from the service provider. A manipulated SAML message should probably not lead to a valid login into the service provider. As standards are not always crystal clear defining the expected result is not always easy and may be subjective. In addition some failed tests may not impact security of a service. Although the test execution API returns a rule value which corresponds to the expected result of a test, this is only a hint. You should always reflect the contents of the test and make your own conclusion of how to weight the test outcome.