a) Create tests earlier.
b) Create more thorough tests.
c) Find starting points for exploratory testing.

Let’s start with the definition of an action. An action consists of the action itself, the preconditions required for the action, and the effects of that action. For this article, we will use logging in as an example. Let’s look at one way of defining a “login” action.

This is a basic login action. In order to log in, you need a valid username and password. If you have that and you log in, you will be authenticated as the user you entered. This simple definition gives us our first test.

Or abstractly, prepare preconditions, perform actions, and verify effects.

There are several things to notice here. First, this test makes no assumptions about the application interface. It is abstract enough that you could run it against a GUI or an API. We will look at what happens when you refine a test to use a specific interface in another article.

The second thing to notice is that the verification is also abstract, too abstract to effectively execute. This is the first place we have found to “test the spec”. Does the application’s specification document explain what it means for a user to be authenticated in enough detail to be able to tell unambiguously that a user is authenticated? If not, it’s time to start asking questions and get that information added to the spec.

This is also the first place where we can “peek inside the box”. What happens under the hood when a user is authenticated? If it’s a web application, is a session cookie sent to the user, and a session started on the server? If it’s a function in an API, are we given an authentication token? These are questions that can both make your test more detailed, and provide a starting point for exploratory testing. Exploratory testing will be covered in a later article.

Now let’s take another look at our login action. The definition given above is a “perfect” test. If we have “perfect” preconditions and we perform the “perfect” action we expect the “perfect” results. What happens if the preconditions are different? Here is the action for an expected failure.

We have taken the two previous preconditions, reversed each of them, and turned them into all of the possible combinations of the preconditions where we expect a failure. If either the username or password is invalid, or both are invalid, and we attempt to login, the user should not be authenticated. This gives us our next tests.

The preconditions and the action are no more complicated than before, but now the verification has a small problem. It is verifying that something does not happen. If we have received our unambiguous definition of what being authenticated means, then on the surface this doesn’t seem difficult. But what happens in the code is another matter. Not only do we as testers often not have the access we need to verify that nothing has changed, but it would be difficult for a developer with full access to everything to demonstrate that nothing has changed in a non-trivial application.

How do we solve this problem? We need more to test. It is possible that absolutely nothing will happen if an invalid login attempt is made, but more likely, there will be a positive testable effect as well. Here is another chance to test the spec; What other effects will occur because of this invalid login attempt? It might look like this.

Now we have something else to verify. We can add the following to the three tests above.

Now we have something testable, but this still doesn’t help us verify point a) of these tests. To do that, we need to use another action. There should be at least one action where “User is not authenticated” is a precondition. Let’s use this one.

For the sake of simplicity, we will use a more concrete action than the ones we have been looking at so far.

If we try to perform this action at the point where we expect the user is not authenticated, then we expect to see an error. If we see the effect, then we know that the precondition is correct assuming that the action is working correctly. If we do not see the expected effect, then we have another problem. Is it the precondition that is wrong or the action that is not working? Still, this gives us a way, uncertain as it may be, to test our original verification. Let’s reorder our tests and add our new steps.

You might be wondering how to pick an action to test the negative effect, or you might be wondering if one action is enough. After all, there will usually be more than one action that uses the precondition we want to test. You’re also depending on the action that you are testing with to work correctly. This is part of the art of testing. Which action do you use, and how many do you use, before you are comfortable that the results are probably accurate? That will be up to your discretion.

This is just the beginning of creating tests for logging in. In the next article we will look at what happens when you combine an action with a concrete interface.

MsgBox(64, "Waiting", "Start the Opera installer.")

AutoItSetOption("SendKeyDelay", 200)

Dim $done
$done = False

While Not $done
	If WinExists("Opera", "has successfully installed") Then
		WinActivate("Opera")
		Send("{SPACE}") ; Close the installer
		$done = true
	ElseIf WinExists("Opera", "Click Install to begin") Then
		WinActivate("Opera")
		Send("{SPACE}") ; Begin the install
	ElseIf WinExists("Opera", "Specify which icons") Then
		WinActivate("Opera")
		Send("{SPACE}") ; Install default icons
	ElseIf WinExists("Opera", "Install Opera") Then
		WinActivate("Opera")
		Send("{TAB}{SPACE}") ; Open the directory dialog
		Send("C:\Opera") ; Change the install directory to C:\Opera
		Send("{ENTER}") ; Close the directory dialog
		Send("{TAB}{TAB}{TAB}{SPACE}") ; Tab to next button and continue
	ElseIf WinExists("Opera", "Custom") Then
		WinActivate("Opera")
		Send("{SPACE}") ; Disable making Opera the default browser
		Send("{TAB}{TAB}") ; Tab to radio buttons
		Send("{DOWN}") ; Select custom install
		Send("{TAB}{TAB}{TAB}{TAB}{SPACE}") ; Tab back to next button and Continue
	ElseIf WinExists("Opera", "agree to this license") Then
		WinActivate("Opera")
		Send("{SPACE}") ; Select I Accept
	ElseIf WinExists("Opera", "Start Setup") Then
		WinActivate("Opera")
		Send("{SPACE}") ; Select Start Setup
	EndIf
WEnd

The basic structure of this style is:

While not done
   If screen 1 text exists then
      do stuff
   Else if screen 2 text exists then
      do stuff
   Else if final screen text exists then
      do stuff
      done=true

Each if statement matches with one screen in the program. Each if statement looks for text or an object unique to a screen, and then executes code to handle that particular screen. On the final screen, you break out of the loop to end your script.

The benefits of this style are:

• No need to code delays. The script keeps looking for the screen that comes after the install is complete.
• It’s quick to write. Manually run through the install once, and note the text on each screen and the steps to be done on each screen.
• It is more maintainable than a script with just a sequence of keystrokes.
• Adding screens only means adding another small block of code.
• It happens to handle a quirk of Windows Installer; each screen is a completely new window.

This style is not good for automating programs with non-linear workflows, such as if the same actions repeat or branch with different data.

That doesn’t sound so bad, but what are the implications of this definition?

the application does what it is supposed to do

If we’re going to test an application, we first have to know what it is supposed to do. Ideally this is answered by other people. Requirements, specifications, and design documents help to answer this question. Common sense should be able to fill in the gaps.

In practice, this never happens. You will always run into a scenario when writing test cases and even while testing where you cannot answer the question, “What is this supposed to do?” Knowing who can answer that question and having an open communication channel to those people is important.

Even with a perfect specification, what an application is supposed to do can be affected by physical limits. For example, a spec may say that an application should support an infinite number of users, but this is practically impossible. Even if you owned every hard drive in the world, there will still be a limited amount of space to store those users in. What an application would really do is, “support a large enough number of users so as to be effectively infinite.” Now you have the question of, “What is ‘a large enough number’?”

making sure that the application does

Once we know what an application is supposed to do, we have to decide how to verify that it does that. This might seem simple, but once again, it proves to be a significant challenge. Consider a program that is a fibonacci sequence generator. A fibonacci sequence is a sequence of numbers where each number in the sequence is the sum of the previous two numbers. The beginning of the sequence goes like this: 1, 1, 2, 3, 5, 8, 13.

Now think about how you would test this. Would you hand calculate the sequence and compare what the program generates to your sequence? If you decide to do that, I hope you really like where you’re sitting. The sequence is infinite, which means you are going to spend the rest of your life trying to test the generator in this way.

Don’t worry. In reality the sequence generator would stop working before that. Unfortunately, it stops working when it crashes. When this happens will depend on multiple factors including the language it was programmed in and what platform it was running on. These affect the implementation of the generator, and it may result in the program crashing when the value passes 4,294,967,296 or 2,147,483,647 or 65,536. Or it may not crash at all and suddenly start spitting out negative numbers. It all depends on how it is implemented in code, and that is generally hidden to testers.

does not do what it is not supposed to do

We already have the problem of testing a program that in theory never ends when it is working correctly. Now we have to consider everything that the sequence generator is not supposed to do. It is not supposed to:

• Output numbers not in the sequence.
• Crash.
• Erase the hard drive.
• Invest your cash in stock in Euro Disney.
• Cause hurricanes.
• Bring you life-long happiness. Sorry.

OK, so the last three are practically impossible, but they are still in the realm of things that the program is not supposed to do. But the first two are reasonable, and even likely, and the third is not as farfetched as it may seem.

Add to this the problem of making sure that something doesn’t happen. This is not always impossible, but it is very difficult. You might find that it crashes when calculating the 5,627th number. You still don’t know if it will crash when calculating the 9,321st number.

The Art of Testing

So by examining our simple definition of testing we’ve gone from difficult, to impossible, to soul-crushingly impossible. This is where the art of testing comes in. A tester’s skill is in covering enough of this infinite space to demonstrate that a user is unlikely to encounter a defect during the time that they use the program. Finding the answers to the question of what an application is supposed to do, figuring how how to make sure it is doing what it is supposed to do, predicting possible points of failure, and finding the failures that nobody ever thought were possible: These are just some of the things that an experienced tester is capable of doing.

Here’s a simple example of a Ruby script using AutoItX. Using AutoItX with Ruby might be useful if you’re using WatiR, and run into an ActiveX control or Java applet.

require 'win32ole'

autoit = WIN32OLE.new('AutoItX3.Control')

if autoit.WinExists("Untitled - Notepad") == 1
	autoit.WinActivate("Untitled - Notepad")
	autoit.Send("Hello world!")
end

Here’s the same example with C#. You have to add a reference to the AutoItX dll first.

using AutoItX3Lib;
/* Other usings and application code */

public void HelloWorld()
{
	AutoItX3Class automator = new AutoItX3Class();

	if (automator.WinExists("Untitled - Notepad", "") == 1)
	{
		automator.WinActivate("Untitled - Notepad", "");
		automator.Send("Hello world!", 0);
	}
}

As this implies, a large focus of this blog will be automated testing and other testing tools. But I also intend to discuss other parts of testing practice. If you don’t know the basic manual practice of testing, no tool in the world is going to make you a good tester. Everything posted here is meant to make you a more effective tester, and to make testing more interesting and rewarding.