Test Data

https://www.flickr.com/photos/tt2times/2568645910
Tony Werman (Flickr.com)

Here are some interesting quotes:

You can’t be Agile if you don’t use the right test data

Developers often copy subsets of production data, sometimes anonymizing it but not always and rarely considering whether the test data contains sufficient diversity to exercise all important edge cases. And those are the more mature dev shops!

The quotes come from two different Forrester articles but are representative of dozens of articles about testing and quality assurance.  But this is not just representative of Forrester, it is representative of the IT industry as a whole.  Let’s dissect each piece and see what we come up with.

You can’t be Agile if you don’t use the right test data

So what is really meant by this?  The concepts around Agile revolve around being able to respond quickly to demands, whether those demands are made before the application is being built or while the application is being built.  In order to deliver applications with a high quality in a timely manner, automated testing is a necessity.  In order to do automated testing appropriately, you need to ensure that you are testing everything that needs to be tested

If you are not using the right test data you cannot guarantee that you are testing everything.  This is vitally important for unit testing.  There are a lot of people who have said in the past that 80% code coverage for unit testing was good.  I’ve even said it.  But it’s not good enough anymore.  100% coverage.  If you’re not going to test it why bother writing it?  If you are going to achieve 100% code coverage you need to know exactly what data you are putting into your system so that you can test everything.  You need the right test data.

Ok, enough said.  Next quote:

Developers often copy subsets of production data, sometimes anonymizing it but not always and rarely considering whether the test data contains sufficient diversity to exercise all important edge cases. And those are the more mature dev shops!

This is the same as the first quote.  Why are you using production test data?  The odds are that it is not going to provide you with 100% code coverage so you are going to need to add data yourself.  And if you keep updating to the latest version of production data you run the risk of losing your 100% code coverage because the data changed.

The only way to ensure that you have 100% code coverage is to create the data that you need.  Enterprise Test Data Management.  It’s tough but eminently doable.  The first company I worked for didn’t allow us to use Production data for our everyday testing, we had to use test data that we created and stored in Generation Data Groups.  This data was the input to our testing stream.  Every time we added new functionality we added new test data.  We added in test data that we knew should never happen in production, but if it did we wanted to handle it.

So, if you say that your project team is Agile ask yourself these two questions:

  1. Do I have 100% code coverage for my unit tests?
  2. Am I using the same data every single time I test?

Two simple questions but they may change how you approach testing.

Leave a Reply