Test Cases

What is a test case

In most cases the understanding of a test case is that it contains a set of instructions to execute to determinate if a software works as specified.

Based on the full traceability in TestResults.io a test case is regarded as a container that includes a test plan and a test report (and revisions thereof).

While a test case contains meta data like name, description and in which test sets it is included, the test plan contains the set of instructions that are used to determinate if a software works as specified. Every time a test plan gets executed the system generates a new test report that is unique to the execution of that test plan and is linked to it.

A test case can contain unlimited revisions of a test plan with each test plan having unlimited test reports.

For every test case only the newest revision of a test plan can be executed.

The Test Cases overview screen lists all test cases registered for the selected project. For every test case you see

• Identifier

• Name

• State of the newest test plan

• Result of the last execution (newest test report of newest test plan)

• Result of the last 10 executions and their duration

Selecting a test case on the overview brings to the Test Case screen:

This screen is divided into four areas that map perfectly to the workflow of managing a test case

Test Case

The test case area shows you the identifier, the name and the description of the selected test case.

Available Operations

Notifications

There are multiple events for a test case that get triggered either by a user action or by the system. You can register for those events to get notified.

In case you are registered for a least one notification the notification flag () will changed to a filled flag: . To see which notifications are currently registered click on the flag. The popup will list all available notifications with the one you have registered for being checked.

Modifying

The Edit Test Case dialog allows you to change the identifier, name an description of a test case.

Printing

If you want to print a short overview of the test case you can use the print () operation to open a specific view of the test case, the selected test plan revision and the selected test report which is easier to print.

In case you want to print the test report, please refer to test report - printing to see alternative options to print a more detailed report.

Archive

Archived test cases are not listed in the default Test Case Overview screen. You need to click on the "Archive" tab on the Test Case Overview screen to see all archived test cases.

To restore a test case from the archive you can either click on any archived test case to see its details and click the restore icon or you can hover over the entry in the archived test cases list and select restore from the options on the right.

Test Plan

The test plan area is where you define the content of your test case, i.e. what should be executed.

Revisions

TestResults.io uses a revision concept for test plans, therefore whenever you need to change a test plan after it was approved you need to create a new revision. This means that every uploaded document, variable or additional information is bound to a specific revision.

The first revision of a test plan is automatically created once a test case is created. Revisions are automatically numbered for you.

To create a new revision you have to click the "NEW PLAN REVISION" button.

The minimum information required for a test plan to be valid is a Test Case Plan Document. As soon as you upload such a document the NEW PLAN REVISION button will change to which allows a test automation engineer to indicate he is working on the automation for this test case.

States

A test plan can have one of the following states

Test Types

TestResults.io supports three different test plan types:

Changing a test plan type, after a test plan was submitted, will create a new revision of the test plan. This new revision will be prefilled with the information from the previous revision and is automatically in the same state as the previous revision.

Document & Supporting Files

The test plan document is a description of what should be tested how. This can either be a structured or free-text document.

For easier maintenance and interoperability we suggest to provide a text document with the following structure:

If you test plan document refers to any additional file that needs to be available to the Subject Under Test you can uploaded those files as Supporting Files. If you want to upload more than one file, upload a zipped archive. The TestResults.io platform will automatically make sure that all files are available to the Subject Under Test.

Execution Variables

In case the test plan requires variables that need to be dynamically set for every execution the test automation engineer will create those variables for you and you can set them by clicking on the "Edit" link.

This will open the "Define execution variables" dialog that allows you to set the value of every available variable.

The TestResults.io platform knows two different kind of variables: Normal and secured.

The names of the variables, as well as if they are secured is defined by the test automation engineer that provided the automated test case.

All variables are pre-populated with values chosen by the test automation engineer. This makes sure that you do not have to provide input after you have uploaded a test plan document.

Test Report

A test report is an executed instance of a specific revision of a test plan. An overview of all reports for a specific test plan is shown in the test plans & reports area:

The result history shows the last 10 executions of the test plan. The bar chart shows the relative execution time in comparison to all other executions of this plan as well as the color coded result.

The test report overview lists all executions with their corresponding execution number, the date they were started and the result state.

The execution number 0 is reserved for currently running executions. This means that all test reports in state "Pending" will have the execution number 0. The execution number is automatically calculated based on the time an execution is finished (in comparison to when it is started). Therefore, a test report might have a higher execution number even-though it was started earlier than another test report with a lower execution number.

Results

A test report can have the following possible execution results:

All results but "Pending" are final, i.e. they cannot be changed afterwards. The result pending is a placeholder till the execution is finished and a final result can be calculated.

Overview Tab

The report details view lists the following details

Detailed Report Tab

Provides you access to all executed steps. In case the test plan was executed more than once, the history will provide you a graphical overview of the duration and the step result of the last 9 executions of this individual step.

By clicking on a screenshot it gets opened in full size.

List of Instances Tab

In case a test report consists of multiple instances, e.g. a performance test plan was executed 5 times in parallel, all instances are listed with their duration and their individual result in the List of Instances tab.

Report States

Independent of its result, a test report can have a state.

While all results but "Pending" are final, the state is changing over the lifetime of a report. Only states marked with an asterisk (*) are final and cannot be changed anymore.

Report Visualizations

If you selected either "performance" or "stability" in your test plan as test plan type, the report shows you specialized visualizations that supports you to determinate if the Subject Under Test fulfills your requirements or -if not- assists you in identifying the root cause.

Performance

For a performance report the TestResults.io portal provides you an additional box plot over all executions.

This box plot allows you to easily identify outliers and to start investigating why those executions took longer than expected. Clicking on an outlier opens its detailed, step-by-step report.

Stability

For a stability report the TestResults.io portal provides you an additional bar chart over all executions.

This bar chart shows the execution time and the result for every instance. This visualization allows you to identify trends within the execution and therefore to pin point typical stability problem patterns.

Click on any instance opens its detailed, step-by-step report.

Printing

TestResults.io creates a printable Microsoft Word Document based on on your template. After downloading the file you can print it with your word processor of choice.

Test Set

The test set are lists all test sets in which the test case is included.

By clicking on a test set you can directly jump to the corresponding test set. By clicking on "ADD TO TEST SET" you can add the test case to a new or existing test set.