Test Cases

Provides information on test case management, including different test types, test plans and test reports

What is a test case

In most cases the understanding of a test case is that it contains a set of instructions to execute to determinate if a software works as specified.

Based on the full traceability in TestResults.io a test case is regarded as a container that includes a test plan and a test report (and revisions thereof).

Test Case, Plan & Remote schematic view

While a test case contains meta data like name, description and in which test sets it is included, the test plan contains the set of instructions that are used to determinate if a software works as specified. Every time a test plan gets executed the system generates a new test report that is unique to the execution of that test plan and is linked to it.

A test case can contain unlimited revisions of a test plan with each test plan having unlimited test reports.

For every test case only the newest revision of a test plan can be executed.

The Test Cases overview screen lists all test cases registered for the selected project. For every test case you see

• Identifier

• Name

• State of the newest test plan

• Result of the last execution (newest test report of newest test plan)

• Result of the last 10 executions and their duration

Selecting a test case on the overview brings to the Test Case screen:

Test Case Screen

This screen is divided into four areas that map perfectly to the workflow of managing a test case

Test Case Screen schematic overview

Test Case

The test case area shows you the identifier, the name and the description of the selected test case.

Available Operations

Image	Operation	Description
	Mark as favorite	Click on the asterisk will toggle if the test case is listed in the favorites section of the projects screen.
	Enable notifications	Opens a menu that allows you to select the event triggers you want to receive a notification for.
	Edit test case information	Opens a dialog to modify the identifier, name or description of the test case.
	Archive test case	Marks the test case as archived. An archived test case can be restored from the archived test cases overview.
	Restore archived test case	Marks the test case as active. Note: This operation is only visible on archived test cases.
	Delete test case	Deletes the test case. This action is not reversible. If you mistakenly deleted a test case please contact support.
	Print test case	Opens a new browser window which shows a printable view of the the test case as well as the currently selected test plan and report.

Notifications

There are multiple events for a test case that get triggered either by a user action or by the system. You can register for those events to get notified.

Trigger	Description
Plan Ready For Review	A new revision of a test plan was uploaded and submitted, i.e. a new revision of the test plan is ready and awaiting automation.
Test Report Completed	A test plan was executed and a test report was created for this execution. Note: This event is triggered whenever a test report was created independent of its result.
Test Report Failed or Aborted	A test plan was executed and a test report was created for this execution which has the result of "failed" or "aborted".

In case you are registered for a least one notification the notification flag () will changed to a filled flag: . To see which notifications are currently registered click on the flag. The popup will list all available notifications with the one you have registered for being checked.

Modifying

Edit Test Case dialog

The Edit Test Case dialog allows you to change the identifier, name an description of a test case.

The identifier can only be changed until at least one test plan of this test case was executed for the first time.

Printing

If you want to print a short overview of the test case you can use the print () operation to open a specific view of the test case, the selected test plan revision and the selected test report which is easier to print.

Printing a test case

In case you want to print the test report, please refer to test report - printing to see alternative options to print a more detailed report.

Archive

Archived test cases are not listed in the default Test Case Overview screen. You need to click on the "Archive" tab on the Test Case Overview screen to see all archived test cases.

To restore a test case from the archive you can either click on any archived test case to see its details and click the restore icon or you can hover over the entry in the archived test cases list and select restore from the options on the right.

Test Plan

The test plan area is where you define the content of your test case, i.e. what should be executed.

Revisions

TestResults.io uses a revision concept for test plans, therefore whenever you need to change a test plan after it was approved you need to create a new revision. This means that every uploaded document, variable or additional information is bound to a specific revision.

The first revision of a test plan is automatically created once a test case is created. Revisions are automatically numbered for you.

To create a new revision you have to click the "NEW PLAN REVISION" button.

The minimum information required for a test plan to be valid is a Test Case Plan Document. As soon as you upload such a document the NEW PLAN REVISION button will change to which allows a test automation engineer to indicate he is working on the automation for this test case.

States

A test plan can have one of the following states

State	Description
New	This revision of the test plan was never submitted for automation. Note: As long as a test plan is in this state all properties can be modified without creating a new revision.
Plan In Review*	The test plan was submitted by the user and is now being checked by an automation engineer. In this phase the engineer checks that he understand the test plan document and that everything that is required to automate the test plan is available. In case the automation engineer is able to accept the test plan, the state of the plan will change to "Automating" and he can start working on the automation. In case the automation engineer does need to reject the test plan, the state will be changed back to "New". This allows the user to update the plan.
Automating	The test plan is currently being automated by an automation engineer.
Review of initial report*	The automation of the test plan is finished and the first execution was done successfully. The results of the first execution need to be accepted by the user to be able to execute the test plan by himself. In case the user accepts the initial report the state is changed to "Ready for execution" which means this test plan can be freely executed.
Ready for execution	This test plan revision can be executed.

States mark with an asterisk (*) are only available if you have the Test Automation Workflow enabled

Test Types

TestResults.io supports three different test plan types:

Type	Description
Default	This is a regular regression test. The test plan is executed once and the results are reported for this specific run.
Performance	The test plan is executed multiple times in parallel. You need to specify the amount of instances that should run at the same time. The TestResults.io platform makes sure that all required environments are ready before the tests are executed. Note: TestResults.io is all about real testing, that means if you want to run a performance test with 100000 clients, the platform spins up 100000 environments.
Stability	The test plan is executed for a given time. If a single execution of the test plan takes less time than your defined minimum execution time the execution is repeated till that minimum time is reached. The unit for the time definition is hours. Info: If you want to run a test for less than an hour you can define fractions of an hour, i.e. 0.1 = 6 minutes.

Changing a test plan type, after a test plan was submitted, will create a new revision of the test plan. This new revision will be prefilled with the information from the previous revision and is automatically in the same state as the previous revision.

Document & Supporting Files

The test plan document is a description of what should be tested how. This can either be a structured or free-text document.

For easier maintenance and interoperability we suggest to provide a text document with the following structure:

Step	Test Input	Expected Result	Comment
1	Click on Start button	The Windows Start menu is displayed	"Startbutton" refers to the default windows start button

There is no limitation in the test plan document, neither in the form nor in the content. It is important that the automation engineer will be able to understand and make sense of the test plan document.

If you test plan document refers to any additional file that needs to be available to the Subject Under Test you can uploaded those files as Supporting Files. If you want to upload more than one file, upload a zipped archive. The TestResults.io platform will automatically make sure that all files are available to the Subject Under Test.

Execution Variables

In case the test plan requires variables that need to be dynamically set for every execution the test automation engineer will create those variables for you and you can set them by clicking on the "Edit" link.

This will open the "Define execution variables" dialog that allows you to set the value of every available variable.

The TestResults.io platform knows two different kind of variables: Normal and secured.

Variable Type	Description
Normal	No special protection of the variable. The content of this variable is visible in the portal to all authorized users.
Secured	A secured variable is not shown in the portal. Once you set the value it is replaced by "(*secured*)" and cannot be recovered from the portal. Note: The automated test case has access to secured variables. Based on the test case design it can be possible that the content of a secured variable is visible within a screen shot.

The names of the variables, as well as if they are secured is defined by the test automation engineer that provided the automated test case.

All variables are pre-populated with values chosen by the test automation engineer. This makes sure that you do not have to provide input after you have uploaded a test plan document.

Test Report

A test report is an executed instance of a specific revision of a test plan. An overview of all reports for a specific test plan is shown in the test plans & reports area:

Reports Overview for selected test plan

The result history shows the last 10 executions of the test plan. The bar chart shows the relative execution time in comparison to all other executions of this plan as well as the color coded result.

The test report overview lists all executions with their corresponding execution number, the date they were started and the result state.

The execution number 0 is reserved for currently running executions. This means that all test reports in state "Pending" will have the execution number 0. The execution number is automatically calculated based on the time an execution is finished (in comparison to when it is started). Therefore, a test report might have a higher execution number even-though it was started earlier than another test report with a lower execution number.

Results

A test report can have the following possible execution results:

Color	State	Description
	Pending*	The test execution was triggered but the report is not finished. Note: This is not a final state.
	Canceled	The test execution was canceled by a user before the report was finished.
	Passed	All test steps of the test plan were successfully executed.
	Failed	One or multiple test steps did not pass because of a functional error in the Subject Under Test. Info: If a test execution is stopped after the first failed test step can be configured in the execution.
	Aborted	One test step was not able to determinate the current state of the Subject Under Test, therefore the test case execution was aborted.
	Error	Either during the initialization or the execution of the test case a non-recoverable error happened in the TestResults.io platform. If you see an error result most of the times there is a problem in the underlying code of the automated test case, therefore you wan to contact your automation engineer to investigate the root cause. For the remote case that you have found a bug in the TestResults.io platform please contact support.

All results but "Pending" are final, i.e. they cannot be changed afterwards. The result pending is a placeholder till the execution is finished and a final result can be calculated.

Overview Tab

The report details view lists the following details

Name	Content
Execution Duration	Aggregated time of all executed test steps. If you have 10 steps with 6 seconds execution time each, the execution duration is 1 minute. If you ran the same test plan as a performance test with 10 instances, the execution duration is 10 minutes.
Average Duration	Average execution time of a single instance. Note: This value is only shown for performance and stability test plans.
Report Status / Overall Status	The current state of the report, see report states.
Execution Result / Overall Result	The aggregated result of all test steps. Info: A test report is only passed if all steps are passed.
Report Instances Count	List the count of total, passed, failed & aborted instances. Note: This value is only shown for performance and stability test plans.

Detailed Report Tab

Provides you access to all executed steps. In case the test plan was executed more than once, the history will provide you a graphical overview of the duration and the step result of the last 9 executions of this individual step.

By clicking on a screenshot it gets opened in full size.

List of Instances Tab

In case a test report consists of multiple instances, e.g. a performance test plan was executed 5 times in parallel, all instances are listed with their duration and their individual result in the List of Instances tab.

List of Instances is only visible for stability and performance test reports.

Report States

Independent of its result, a test report can have a state.

State	Description
Queued	The current report was registered in the system and is currently waiting to be picked up by scaler.
Preparing	The report was picked up by a scaler and the test environment is currently prepared.
Running	The test plan is currently being executed.
Canceled*	The execution was canceled by the users before a final result was calculated.
Completed	The test report is completed and data is currently be collected.
In Review	The test report is finished and ready for review.
Accepted*	The test report was accepted by a user with the corresponding permissions.
In Investigation	The test report was rejected by the user and is currently investigated by the test automation engineer. Note: If the test automation engineer finds an error in the execution the state will be changed to "Engine Execution Failure", otherwise the state is reverted to "In Review".
Engine Execution Failure*	The result of a test report was rejected by the user and the test automation engineer found a problem in the execution.

While all results but "Pending" are final, the state is changing over the lifetime of a report. Only states marked with an asterisk (*) are final and cannot be changed anymore.

Report Visualizations

If you selected either "performance" or "stability" in your test plan as test plan type, the report shows you specialized visualizations that supports you to determinate if the Subject Under Test fulfills your requirements or -if not- assists you in identifying the root cause.

Performance

For a performance report the TestResults.io portal provides you an additional box plot over all executions.

Performance box plot

This box plot allows you to easily identify outliers and to start investigating why those executions took longer than expected. Clicking on an outlier opens its detailed, step-by-step report.

Stability

For a stability report the TestResults.io portal provides you an additional bar chart over all executions.

Stability bar chart

This bar chart shows the execution time and the result for every instance. This visualization allows you to identify trends within the execution and therefore to pin point typical stability problem patterns.

Click on any instance opens its detailed, step-by-step report.

Printing

TestResults.io creates a printable Microsoft Word Document based on on your template. After downloading the file you can print it with your word processor of choice.

Test Set

The test set are lists all test sets in which the test case is included.

By clicking on a test set you can directly jump to the corresponding test set. By clicking on "ADD TO TEST SET" you can add the test case to a new or existing test set.