Область тестирования

The scope of a test defines what areas of a customer's product are supposed to get tested, what functionalities to focus on, what bug types the customer is interested in, and what areas or features should not be tested by any means.

If something is in scope, please test it; if something is out of scope, it should not be tested. Understanding the scope of a test is crucial to be a successful tester on our platform.

What information makes up the scope of a test?

Test environment

First things first – The URL given in section Access at the top of the test overview page defines what website or app should be tested. Any other websites or apps are not under test (unless stated differently in the test description). For more information, see Testing Environment.

You can only see access information once the test started (mind the countdown at the top).

Features

Each test contains at least one of what we call features. A feature can describe an area of a product, e.g. a landing page, product overview page or product detail page of a webshop, it can describe a process like the checkout process of a website or an app, or it can even be a particular functionality that should be tested. For example:

When a customer sets up a new test, so-called standard features will be added by default. Customers can use these standard features and modify them if they want or add completely new features.

You can only submit bugs if a corresponding feature exists in the test. For example, if none of the given features covers the checkout process, you cannot submit bugs related to it.

Bug types and severities

Customers are usually only interested in certain bug types and/or severities, not in all of them at once (Functional, Content, Visual, Usability). You can see which bug types and severities are in scope by checking out the payout table in the right sidebar on the test overview page:

If a bug type/severity is not listed there, you won't be able to select it in the bug form, and it is thus out of scope. E.g. if the payout table does not list the bug type Visual, you may not report visual bugs.

Test instructions

Test instructions are displayed below the Access section and they consist of three sections:

Goal of this test: This section usually covers a general purpose of the test. It often names what to focus on, e.g. new feature that is about to be released.
Out of scope: What is not supposed to get tested is specifically stated here. On live websites, you may oftentimes not complete orders at the end of a checkout process, or some areas of a website or an app might not be ready yet. Make sure to stick to this information – disregarding them can have big consequences.
Additional requirements: If there is anything else to point out, information will be given in this section. This might be information concerning the whole test, credentials for user accounts, dummy payment information, etc.

Attachments

In rare cases, one or more attachments might be provided. Those are usually Microsoft Excel spreadsheets or PDF files given by the customer directly.

Requested devices

You will usually be invited for a specific device of yours. This device is displayed in the right sidebar on the test overview page. For an ideal device coverage, our distribution algorithm invites testers for different devices. You may only use the requested devices that were assigned to you.

If the seats for the requested device are taken, you can join the test with a different device. On the 'Submit a Bug' page you can check with which device you are in the test.

Chat information

The team leader or Customer Success Manager might have provided additional information about the scope or posted important reminders. Please always check the chat if you see that there are new messages.

Out of scope

Out of scope means that something is not supposed to be tested. Note that the following points are always out of scope (unless stated differently in the test description):

Legal problems. We are not legal advisors and the severity of a bug is not determined by legal provisions, frameworks, or standards.
Problems related to smart filters or virus scanners, e.g. blocking the execution of apps.
Setup problems in tests.

Setup problems

Setup problems are not legitimate bugs – it only means there is a problem with the test setup or the test environment. Please report them to the team leader via the test chat who will in turn contact the CSM. Here are a few examples:

The URL given in the test environment section of a test is wrong (404).
The test environment can only be accessed with credentials, but none are given or the given ones don't work.
An app cannot be downloaded or installed.
The customer provided login credentials for user accounts but they don't work.
A link on a staging environment redirects to their live website. The link was not updated to point to the staging environment during the test environment setup.
The device requirements for a test e.g. list iOS 9 but the app under test can only be installed on iOS 10 and above.

The scope is unclear or contains contradicting information

If you don't understand something about the scope or information contradict each other, please contact the test's team leader via the test chat as described in article Confusing Instructions.

Sharing such information helps us improve setups of future tests.

Why is information sometimes not consistent?

When customers intend to run a new test but the test setup of a previous test can be reused for the most part, they will often duplicate that old test and adjust severities as well as additional requirements. Features are usually not adjusted, so sometimes irrelevant ones are available or feature descriptions and additional requirements contradict each other. Since features were duplicated but additional requirements were specifically written for the new test, additional requirements often override feature descriptions.

If you spot contradicting information, please ask the team leader via the chat rather than making assumptions.