Test Sets
Provides information on Test Sets and their execution
A test set contains any number of test cases and allows the user to classify test cases into user defined categories. Test sets are also used as a starting point to execute test cases.
The Test Sets overview screen lists all test tests registered for the selected project. For every test set you see
  • Name
  • Last Execution date and time
  • The software version that was used for the last execution
  • The pass rate of all included test cases
Selecting a test set on the overview brings to the Test Set screen:
Test Set Screen
Like the test case screen, the test set screen is divided into four major areas:
Test Set Screen schematic view

Test Set

The test set area shows you the identifier, the name and the description of the selected test set.

Available Operations

Image
Operation
Description
Mark as favorite
Click on the asterisk will toggle if the test set is listed in the favorites section of the projects screen.
Enable notifications
Opens a menu that allows you to select the event triggers you want to receive a notification for.
Edit test set information
Opens a dialog to modify the identifier, name or description of the test set.
Archive test set
Marks the test set as archived. An archived test set can be restored from the archived test sets overview.
Restore archived test case
Marks the test set as active.
Note: This operation is only visible on archived test sets.
Delete test set
Deletes the test set. This action is not reversible. If you mistakenly deleted a test case please contact support.

Notifications

There are multiple events for a test case that get triggered either by a user action or by the system. You can register for those events to get notified.
Trigger
Description
Execution completed
An execution run was completed and results are available
Test Report failed or aborted
A test report is finished with the result FAILED or ABORTED
In case you are registered for a least one notification the notification flag (
) will changed to a filled flag:
. To see which notifications are currently registered click on the flag. The popup will list all available notifications with the one you have registered for being checked
.

Modifying

Edit Test Set dialog
The Edit Test Set dialog allows you to change the name and description of a test set.

Archive

Archived test sets are not listed in the default Test Set Overview screen. You need to click on the "Archive" tab on the Test Set Overview screen to see all archived test sets.
To restore a test set from the archive you can either click on any archived test set to see its details and click the restore icon
or you can hover over the entry in the archived test sets list and select restore from the options on the right.

Test Cases

The test cases lists all test cases that are currently included in this set. In addition it allows you to add or remove test cases to and from the selected test set.
The overview lists the following attributes of any included test case:

Add/Remove test cases

By clicking on the "MANAGE TEST CASES" button the "Included Test Cases in Test Set" dialog is shown.
This dialog shows you all test cases in the current project. If checked, a test case is included in the current test set, if not checked a test case is not included in the current test set.
To open a test case in a new window to see even more details you click on the OPEN button (
).
By clicking on a column label, the column will be sorted. By clicking on the filter icon the filter dialog will be shown, which allows to search for specific test sets

Add by test set

An alternative way to included test cases in a test set is by copying existing test sets. By clicking on the "ADD BY TEST SET" button you can select an existing test set which is copied.
Click on the "SAVE" button to copy a selected test set into the current test set.
Important: The test set is copied, not referenced. Changing the original test set after you have copied it, does not impact the new test set.

Result History

The result history provides you a quick overview of the number of executed test cases included in this test set per software version. This gives you a good indication if your test coverage is increasing, decreasing or in a steady state.
In addition the history area distinguishes between passed, failed, canceled and aborted test sets. This allows you to see the quality of a specific software version.

Visualizations

Icon
Description
Shows an area graph of the number of test cases per software version and the result state.
Shows individual bar graphs of the number of test cases per software version and the result state.

Filtering

You can filter the result history by test environment, date range or tags that you have used for executions.
To reset the filter, clear all fields and click on the "FILTER" button.

Executions

The executions area gives you an overview of all executions of this test set. For every test set you see
  • Completion time - the time the last result of the last test report was received
  • Software version with which the execution was run
  • Test environment on which the execution was run
  • User who requested the execution
  • All tags used for the execution
  • Overall result of all test reports created by this execution
By clicking on an individual execution result a new tab is opened. This tab lists all test cases that where executed with the specification execution.
For every included test case the following details are shown
  • Test case identifier
  • Test case name
  • Revision - test plan revision
  • Execution No - test report revision
  • Test report status
  • Test report result
  • Time the test report was finished
  • Execution time

Execute a test set

A test set can be executed by clicking on the "EXECUTE" button, which opens the "Execute Test Set" dialog.
The execute test set dialog allows you to select a test environment on which the Subject Under Test should run, as well as a software version that you want to run the tests against.
For easier identification you can define freely chosen tags for the execution.

Control tags

The TestResults.io platform supports some predefined control tags. Do not use any of the following reserved tags until asked for by the TestResults.io support or your vendor:
  • -sim:
  • -debug:
  • -livetrack
Please contact support for additional information on these specific control tags.
An execution is started by clicking on the "EXECUTE" button.