To understand the health of the code in QUBO algorithm, we needed to understand the performance of the algorithm. We ran a sanity test on each check-in and then a more comprehensive test suite each night there was a check-in. The reporting provided Azure DevOps Analytics for build / tests provided was not enough to evaluate the algorithm performance, so we augmented the information from the tests and created this report. information from a test runner point of view.

To understand the health of the code in QUBO algorithm, we needed to understand the performance of the algorithm. We ran a sanity test on each check-in and then a more comprehensive test suite each night there was a check-in. The reporting provided Azure DevOps Analytics for build / tests provided was not enough to evaluate the algorithm performance, so we augmented the information from the tests and created this report.

What questions are answered?

What is the health of the tests for each pipeline ?

For each test, how long did the algorithm take to run ?

What is the trend for each test run ?

Did the algorithm find the correct answer for this test ?

If a test failed, was it because of a incorrect result or from maximum time allowed for the test ?

What is the history for a specific test (using Q&A) ?

When did the last test suite fail (using Q&A) ?

How was this implemented?

Each test is an execution of an algorithm with a given testset. The custom data captured is the input to the algorithm, amount of time the algorithm is allowed to run, actual run time for the algorithm, and the result provided by the algorithm.

This custom data is written an Azure Blob in .csv format. An Azure Pipeline then pushes the test run information (.trx) into to an Azure blob. The files in the Azure Blob are organized by the build id and the name of the pipeline.

Finally, the .pbix on each refresh reads the files in the Azure Blob into a semantic model and the updates the reports and dashboards.

What next?

What is the environment for each test (cpu, memory, gpu characteristics) ?