Test Automation Workflow

Provides information on the process that describes the steps from an uploaded test plan document to a fully automated test case and its reporting.

As TestResults.io is a automated testing platform it is target for individual testers, companies with multiple internal automation teams as well as external service providers who provide automation services to their clients.

In addition, all of the provide services need to be fully traceable, e.g. revisions need to be tracked, and repeatable (see regulated markets).

To make sure that individual testers don't get overloaded, TestResults.io provides to option to disable the Test Automation Workflow. Please refer to the section Automating a test case to understand the differences in your responsibilities if you disable the Test Automation Workflow.

The Test Automation Workflow consists of two connected workflows: The test plan workflow and the test report workflow.

Test Plan Workflow

This workflow starts with the user creating a new test plan revision. Once the user has confirmed that all required information is provided the test plan is submitted for design.

After submission, the automation engineer reviews the available information and makes sure that everything is understood. If the automation engineer is missing information, even after clarification, he can reject the test plan, which allows the user to provide more information. In case the provided information allows the automation engineer to start automating the test plan the test plan is accepted.

After automation of the test plan is finished and confirmed to work, the automation engineer runs an initial execution (execution 0). If execution 0 finishes as expected the automation engineer claims that the automation of the test plan is done.

The user can verify that the test report of execution 0 fulfills the requirements and accepts the test report or requests a redesign with specific comments on the requested changes.

Acceptance of the test report of execution 0 put the test plan in the state "Ready for Execution" and the user can start this test.

Test Report Workflow

This workflow start with the user requesting an execution of a test set. When the user executes a test set an execution gets queued and a test report gets generated with revision 0. Revision 0 is a placeholder for the actual revision with the actual revision being defined at the time when the test plan is running.

Once a scaler picks the execution up and schedules it for execution, the test report is in the state Preparing. As soon as the test environment is ready and the Subject Under Test is available on the target system the system starts to run the automated test plan.

At any of the states queued, preparing & running the user can cancel the execution. If the user cancels the execution after it was scheduled for its execution (State: Preparing) it results in an canceled test report revision. If the user cancels while the execution is still queued, no test report revision is created.

A running execution always results in a test report. In case the execution is terminated because of a system error (hosting problem, execution engine problem, test automation problem) the report is marked with Execution Error. In all other cases the test report can be reviewed. The user can either accept the test report or reject it. While n acceptance of the test report finishes the work flow, a rejection triggers an Investigation by the automation engineer.

If the automation engineer identifies that the problem lies within the automation itself the test report gets marked with Error in Execution. In case the problem is based because of a malfunction in the Subject Under Test or an error in the test plan itself the automation engineer adds this information to the test report and sets the test report back to ready for review.

For revision security a test report can only be rejected within 15 days. If not rejected within 15 days the system treats the test report as accepted without marking it as accepted. A test report can always be accepted by the user, even after the 15 days security margin for rejecting it.

For regulatory reasons (FDA electronic records, etc.) a test report can not be automatically accepted.

Connections between Test Plan and Test Report Workflow

The Test Plan Workflow and the Test Report Workflow interact in two points point.

Execution Failure

In case on an execution failure the TestResults.io portal's active monitoring system will automatically reset the plan status to "Automating". This automatic interaction between the two workflows blocks the user from triggering new execution until the underlying problem is solved.

Ready for Execution

The Test Report Workflow depends on the Test Plan Workflow as its initial state is connected to the last state of the Test Plan Workflow. A test report can only be generated if the test plan is in Ready for Execution state.

Complete Workflow