Test Automation


David Travis

So the next step in our “Accelerate” journey is Test Automation.

It sounds simple, doesn’t it?  Put a couple of scripts together, do some testing via a batch job and BAM you’re done.  Walk away into the sunset, with the sound of your spurs the only thing that can be heard for miles.

Too bad that this image is just a fantasy.  Simple it is not.  Priceless it is.

There are two key factors to having good, effective tests that can be automated:

  • Reliable and that includes repeatable (see tomorrow’s note for details)
  • Written by the proper people

Let’s take a look at reliable first.  When you think of a reliable car you think of numerous traits like the ability start in cold weather, not having issues with accelerating or breaking and not having those annoying warning lights pop up on you for all sorts of stupid reasons.  A reliable test is also the same way in that it runs when you need it to, it produces the results that match the test and, most importantly, it doesn’t generate false positives.

If you’ve got a test that works most of the time then you have a test which is useless all of the time.  How do you know when there is a false positive?  Or a false negative?  You don’t, so you need to investigate the result.  So why did you even run the test if you can’t trust the results?

Written by the proper people is one that is going to be difficult for some people to hear.  The best people to write automated tests are actually the developers, not the QA team.  The authors of Accelerate stated it this way:

The theory behind this is that when developers are involved in creating and maintaining acceptance tests, there are two important effects.  First the code becomes more testable when developers write tests.  This is one of the main reasons why test-drive development is an important practice – It forces developer to create more testable designs.  Second, when developers are responsible for the automated tests, they care more about them and will invest more effort into maintaining and fixing them.

So, is there a need for a QA team, a dedicated testing team?  Yes there is.  Exploratory testing, acceptance testing and usability testing are key things that are uniquely human in what needs to be done.  And let’s not forget that while the developers are creating the tests, they are most likely doing it in conjunction with the testing team.  Testers will approach the application from a completely different perspective from the developer and it is this perspective that needs to be automated.

What now?

Now that you’ve decided to go down the automated testing road you need to figure out when to do it.  And that answer is as often as possible.  The most suitable spot for all of this testing is when you do a commit.  Whenever code is added to the branch / trunk you compile the code and then run the automated testing.

How many tests should you have?  1.6 tests per 10 lines of code.  Just kidding.  There is no right or wrong number.  Two teams could approach the same code and create a different number tests.  The number of tests you need to create is the exact number that you need to feel comfortable that the code works as expected and is safe to move into production.  If you’re not comfortable add more tests until you are.  But don’t add too many.  Like the code that is running in the application you need to maintain the tests as well so adding tests that provide no benefit does nothing more than slow you down.

But what do you do about data?  Stay tuned for more tomorrow.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.