Designing Tests With Actions – Part 1

There are many ways to design tests. In this article we will look at how to design tests by breaking an application down into actions. Designing a test case based on actions allows you to create tests at multiple layers of abstraction, which helps let you:

a) Create tests earlier.
b) Create more thorough tests.
c) Find starting points for exploratory testing.

Let’s start with the definition of an action. An action consists of the action itself, the preconditions required for the action, and the effects of that action. For this article, we will use logging in as an example. Let’s look at one way of defining a “login” action.

Login
Preconditions	Effects
Valid username Valid password	User is authenticated

This is a basic login action. In order to log in, you need a valid username and password. If you have that and you log in, you will be authenticated as the user you entered. This simple definition gives us our first test.

Or abstractly, prepare preconditions, perform actions, and verify effects.

There are several things to notice here. First, this test makes no assumptions about the application interface. It is abstract enough that you could run it against a GUI or an API. We will look at what happens when you refine a test to use a specific interface in another article.

The second thing to notice is that the verification is also abstract, too abstract to effectively execute. This is the first place we have found to “test the spec”. Does the application’s specification document explain what it means for a user to be authenticated in enough detail to be able to tell unambiguously that a user is authenticated? If not, it’s time to start asking questions and get that information added to the spec.

This is also the first place where we can “peek inside the box”. What happens under the hood when a user is authenticated? If it’s a web application, is a session cookie sent to the user, and a session started on the server? If it’s a function in an API, are we given an authentication token? These are questions that can both make your test more detailed, and provide a starting point for exploratory testing. Exploratory testing will be covered in a later article.

Now let’s take another look at our login action. The definition given above is a “perfect” test. If we have “perfect” preconditions and we perform the “perfect” action we expect the “perfect” results. What happens if the preconditions are different? Here is the action for an expected failure.

Login
Preconditions	Effects
Valid username and invalid password OR invalid username and a valid password OR invalid username and invalid password	User is not authenticated

We have taken the two previous preconditions, reversed each of them, and turned them into all of the possible combinations of the preconditions where we expect a failure. If either the username or password is invalid, or both are invalid, and we attempt to login, the user should not be authenticated. This gives us our next tests.

The preconditions and the action are no more complicated than before, but now the verification has a small problem. It is verifying that something does not happen. If we have received our unambiguous definition of what being authenticated means, then on the surface this doesn’t seem difficult. But what happens in the code is another matter. Not only do we as testers often not have the access we need to verify that nothing has changed, but it would be difficult for a developer with full access to everything to demonstrate that nothing has changed in a non-trivial application.

How do we solve this problem? We need more to test. It is possible that absolutely nothing will happen if an invalid login attempt is made, but more likely, there will be a positive testable effect as well. Here is another chance to test the spec; What other effects will occur because of this invalid login attempt? It might look like this.

Login
Preconditions	Effects
Valid username and invalid password OR invalid username and a valid password OR invalid username and invalid password	User is not authenticated An error occurs indicating that the username or password is invalid

Now we have something else to verify. We can add the following to the three tests above.

Now we have something testable, but this still doesn’t help us verify point a) of these tests. To do that, we need to use another action. There should be at least one action where “User is not authenticated” is a precondition. Let’s use this one.

View a restricted web page
Preconditions	Effects
User is not authenticated	An error occurs indicating that the user is not authenticated

For the sake of simplicity, we will use a more concrete action than the ones we have been looking at so far.

If we try to perform this action at the point where we expect the user is not authenticated, then we expect to see an error. If we see the effect, then we know that the precondition is correct assuming that the action is working correctly. If we do not see the expected effect, then we have another problem. Is it the precondition that is wrong or the action that is not working? Still, this gives us a way, uncertain as it may be, to test our original verification. Let’s reorder our tests and add our new steps.

You might be wondering how to pick an action to test the negative effect, or you might be wondering if one action is enough. After all, there will usually be more than one action that uses the precondition we want to test. You’re also depending on the action that you are testing with to work correctly. This is part of the art of testing. Which action do you use, and how many do you use, before you are comfortable that the results are probably accurate? That will be up to your discretion.

This is just the beginning of creating tests for logging in. In the next article we will look at what happens when you combine an action with a concrete interface.