Test Cases
Provides information on test case management, including different test types, test plans and test reports
What is a test case
In most cases the understanding of a test case is that it contains a set of instructions to execute to determinate if a software works as specified.
Based on the full traceability in TestResults.io a test case is regarded as a container that includes a test plan and a test report (and revisions thereof).
While a test case contains meta data like name, description and in which test sets it is included, the test plan contains the set of instructions that are used to determinate if a software works as specified. Every time a test plan gets executed the system generates a new test report that is unique to the execution of that test plan and is linked to it.
A test case can contain unlimited revisions of a test plan with each test plan having unlimited test reports.
For every test case only the newest revision of a test plan can be executed.
The Test Cases overview screen lists all test cases registered for the selected project. For every test case you see
Identifier
Name
State of the newest test plan
Result of the last execution (newest test report of newest test plan)
Result of the last 10 executions and their duration
Selecting a test case on the overview brings to the Test Case screen:
This screen is divided into four areas that map perfectly to the workflow of managing a test case
Test Case
The test case area shows you the identifier, the name and the description of the selected test case.
Available Operations
Image | Operation | Description |
Mark as favorite | Click on the asterisk will toggle if the test case is listed in the favorites section of the projects screen. | |
Enable notifications | Opens a menu that allows you to select the event triggers you want to receive a notification for. | |
Edit test case information | Opens a dialog to modify the identifier, name or description of the test case. | |
Archive test case | Marks the test case as archived. An archived test case can be restored from the archived test cases overview. | |
Restore archived test case | Marks the test case as active. Note: This operation is only visible on archived test cases. | |
Delete test case | Deletes the test case. This action is not reversible. If you mistakenly deleted a test case please contact support. | |
Print test case | Opens a new browser window which shows a printable view of the the test case as well as the currently selected test plan and report. |
Notifications
There are multiple events for a test case that get triggered either by a user action or by the system. You can register for those events to get notified.
Trigger | Description |
Plan Ready For Review | A new revision of a test plan was uploaded and submitted, i.e. a new revision of the test plan is ready and awaiting automation. |
Test Report Completed | A test plan was executed and a test report was created for this execution. Note: This event is triggered whenever a test report was created independent of its result. |
Test Report Failed or Aborted | A test plan was executed and a test report was created for this execution which has the result of "failed" or "aborted". |
Modifying
The Edit Test Case dialog allows you to change the identifier, name an description of a test case.
The identifier can only be changed until at least one test plan of this test case was executed for the first time.
Printing
In case you want to print the test report, please refer to test report - printing to see alternative options to print a more detailed report.
Archive
Archived test cases are not listed in the default Test Case Overview screen. You need to click on the "Archive" tab on the Test Case Overview screen to see all archived test cases.
Test Plan
The test plan area is where you define the content of your test case, i.e. what should be executed.
Revisions
TestResults.io uses a revision concept for test plans, therefore whenever you need to change a test plan after it was approved you need to create a new revision. This means that every uploaded document, variable or additional information is bound to a specific revision.
The first revision of a test plan is automatically created once a test case is created. Revisions are automatically numbered for you.
To create a new revision you have to click the "NEW PLAN REVISION" button.
States
A test plan can have one of the following states
State | Description |
New | This revision of the test plan was never submitted for automation. Note: As long as a test plan is in this state all properties can be modified without creating a new revision. |
Plan In Review* | The test plan was submitted by the user and is now being checked by an automation engineer. In this phase the engineer checks that he understand the test plan document and that everything that is required to automate the test plan is available. In case the automation engineer is able to accept the test plan, the state of the plan will change to "Automating" and he can start working on the automation. In case the automation engineer does need to reject the test plan, the state will be changed back to "New". This allows the user to update the plan. |
Automating | The test plan is currently being automated by an automation engineer. |
Review of initial report* | The automation of the test plan is finished and the first execution was done successfully. The results of the first execution need to be accepted by the user to be able to execute the test plan by himself. In case the user accepts the initial report the state is changed to "Ready for execution" which means this test plan can be freely executed. |
Ready for execution | This test plan revision can be executed. |
States mark with an asterisk (*) are only available if you have the Test Automation Workflow enabled
Test Types
TestResults.io supports three different test plan types:
Type | Description |
Default | This is a regular regression test. The test plan is executed once and the results are reported for this specific run. |
Performance | The test plan is executed multiple times in parallel. You need to specify the amount of instances that should run at the same time. The TestResults.io platform makes sure that all required environments are ready before the tests are executed. Note: TestResults.io is all about real testing, that means if you want to run a performance test with 100000 clients, the platform spins up 100000 environments. |
Stability | The test plan is executed for a given time. If a single execution of the test plan takes less time than your defined minimum execution time the execution is repeated till that minimum time is reached. The unit for the time definition is hours. Info: If you want to run a test for less than an hour you can define fractions of an hour, i.e. 0.1 = 6 minutes. |
Changing a test plan type, after a test plan was submitted, will create a new revision of the test plan. This new revision will be prefilled with the information from the previous revision and is automatically in the same state as the previous revision.
Document & Supporting Files
The test plan document is a description of what should be tested how. This can either be a structured or free-text document.
For easier maintenance and interoperability we suggest to provide a text document with the following structure:
Step | Test Input | Expected Result | Comment |
1 | Click on Start button | The Windows Start menu is displayed | "Startbutton" refers to the default windows start button |
There is no limitation in the test plan document, neither in the form nor in the content. It is important that the automation engineer will be able to understand and make sense of the test plan document.
If you test plan document refers to any additional file that needs to be available to the Subject Under Test you can uploaded those files as Supporting Files. If you want to upload more than one file, upload a zipped archive. The TestResults.io platform will automatically make sure that all files are available to the Subject Under Test.
Execution Variables
In case the test plan requires variables that need to be dynamically set for every execution the test automation engineer will create those variables for you and you can set them by clicking on the "Edit" link.
This will open the "Define execution variables" dialog that allows you to set the value of every available variable.
The TestResults.io platform knows two different kind of variables: Normal and secured.
Variable Type | Description |
Normal | No special protection of the variable. The content of this variable is visible in the portal to all authorized users. |
Secured | A secured variable is not shown in the portal. Once you set the value it is replaced by "(*secured*)" and cannot be recovered from the portal. Note: The automated test case has access to secured variables. Based on the test case design it can be possible that the content of a secured variable is visible within a screen shot. |
The names of the variables, as well as if they are secured is defined by the test automation engineer that provided the automated test case.
All variables are pre-populated with values chosen by the test automation engineer. This makes sure that you do not have to provide input after you have uploaded a test plan document.
Test Report
A test report is an executed instance of a specific revision of a test plan. An overview of all reports for a specific test plan is shown in the test plans & reports area:
The result history shows the last 10 executions of the test plan. The bar chart shows the relative execution time in comparison to all other executions of this plan as well as the color coded result.
The test report overview lists all executions with their corresponding execution number, the date they were started and the result state.
The execution number 0 is reserved for currently running executions. This means that all test reports in state "Pending" will have the execution number 0. The execution number is automatically calculated based on the time an execution is finished (in comparison to when it is started). Therefore, a test report might have a higher execution number even-though it was started earlier than another test report with a lower execution number.
Results
A test report can have the following possible execution results:
Color | State | Description |
Pending* | The test execution was triggered but the report is not finished. Note: This is not a final state. | |
Canceled | The test execution was canceled by a user before the report was finished. | |
Passed | All test steps of the test plan were successfully executed. | |
Failed | One or multiple test steps did not pass because of a functional error in the Subject Under Test. Info: If a test execution is stopped after the first failed test step can be configured in the execution. | |
Aborted | One test step was not able to determinate the current state of the Subject Under Test, therefore the test case execution was aborted. | |
Error | Either during the initialization or the execution of the test case a non-recoverable error happened in the TestResults.io platform. If you see an error result most of the times there is a problem in the underlying code of the automated test case, therefore you wan to contact your automation engineer to investigate the root cause. For the remote case that you have found a bug in the TestResults.io platform please contact support. |
All results but "Pending" are final, i.e. they cannot be changed afterwards. The result pending is a placeholder till the execution is finished and a final result can be calculated.
Overview Tab
The report details view lists the following details
Name | Content |
Execution Duration | Aggregated time of all executed test steps. If you have 10 steps with 6 seconds execution time each, the execution duration is 1 minute. If you ran the same test plan as a performance test with 10 instances, the execution duration is 10 minutes. |
Average Duration | Average execution time of a single instance. Note: This value is only shown for performance and stability test plans. |
Report Status / Overall Status | The current state of the report, see report states. |
Execution Result / Overall Result | The aggregated result of all test steps. Info: A test report is only passed if all steps are passed. |
Report Instances Count | List the count of total, passed, failed & aborted instances. Note: This value is only shown for performance and stability test plans. |
Detailed Report Tab
Provides you access to all executed steps. In case the test plan was executed more than once, the history will provide you a graphical overview of the duration and the step result of the last 9 executions of this individual step.
By clicking on a screenshot it gets opened in full size.
List of Instances Tab
In case a test report consists of multiple instances, e.g. a performance test plan was executed 5 times in parallel, all instances are listed with their duration and their individual result in the List of Instances tab.
List of Instances is only visible for stability and performance test reports.
Report States
Independent of its result, a test report can have a state.
State | Description |
Queued | The current report was registered in the system and is currently waiting to be picked up by scaler. |
Preparing | The report was picked up by a scaler and the test environment is currently prepared. |
Running | The test plan is currently being executed. |
Canceled* | The execution was canceled by the users before a final result was calculated. |
Completed | The test report is completed and data is currently be collected. |
In Review | The test report is finished and ready for review. |
Accepted* | The test report was accepted by a user with the corresponding permissions. |
In Investigation | The test report was rejected by the user and is currently investigated by the test automation engineer. Note: If the test automation engineer finds an error in the execution the state will be changed to "Engine Execution Failure", otherwise the state is reverted to "In Review". |
Engine Execution Failure* | The result of a test report was rejected by the user and the test automation engineer found a problem in the execution. |
While all results but "Pending" are final, the state is changing over the lifetime of a report. Only states marked with an asterisk (*) are final and cannot be changed anymore.
Report Visualizations
If you selected either "performance" or "stability" in your test plan as test plan type, the report shows you specialized visualizations that supports you to determinate if the Subject Under Test fulfills your requirements or -if not- assists you in identifying the root cause.
Performance
For a performance report the TestResults.io portal provides you an additional box plot over all executions.
This box plot allows you to easily identify outliers and to start investigating why those executions took longer than expected. Clicking on an outlier opens its detailed, step-by-step report.
Stability
For a stability report the TestResults.io portal provides you an additional bar chart over all executions.
This bar chart shows the execution time and the result for every instance. This visualization allows you to identify trends within the execution and therefore to pin point typical stability problem patterns.
Click on any instance opens its detailed, step-by-step report.
Printing
TestResults.io creates a printable Microsoft Word Document based on on your template. After downloading the file you can print it with your word processor of choice.
Test Set
The test set are lists all test sets in which the test case is included.
By clicking on a test set you can directly jump to the corresponding test set. By clicking on "ADD TO TEST SET" you can add the test case to a new or existing test set.
Last updated
