Impact of Test Data on Test Automation

What can make a good test a weak one? What can make a simple test an effective one?

Data.

Or Test Data to be precise.

A test and its test-data are like body and soul. But like in real life, soul often gets neglected. While test developers focus a lot of their energies in automating most complex of test scenarios, same extent of attention often gets denied to test data. This neglect can manifest in different ways –

Common pit-falls in test data implementation

 a. Didn’t know, didn’t do

This means the test developer was not aware of the significance of test data and missed to implement any special strategy for the same.

 b. Strong Coupling with Logic

Another mistake which happens is tight coupling with test logic. The flow gets intermingled with setting/reading of data and gets strongly embedded in the DNA of the test. It becomes difficult to see test separately from its data. For ex, test to check if account gets locked after multiple repeated attempts can suffer from this issue when the number of attempts is hard-coded in the test. In future when this business condition changes, test becomes invalid. Sometimes changing such hard-coded data has severe impact on the over-all test and results may get misinterpreted.

 c. Randomize Test Data

Quite often test data is left to random data generators. While this can be useful in certain stress testing scenarios, it can be counter-productive to rich test executions which require meaningful data.

 d. Repeated Data

Each test run gets executed with same test data every time. This monotonous repeat execution may fail to detect any bugs which are associated with usage of data since some of the data-flows may never get tested.

 So what do we do? Few tips from our Experience…

So what should be the approach for proper data handling in test automation scenarios? Well, the key lies in moderation and mix-match of approaches.

a. Give importance to test data

First and foremost, give that importance to test data it deserves. Invest in test-data planning when you plan for test automation scenarios.

 b. Separation of Concerns

Keep test logic separate from the data. Rather than testing for 3 attempts, test for n attempts where n could be picked based on a test strategy.

 c. Randomize with caution.

Instead of having a randomizer as sole test data provider, implement a test data strategy factory. This will provide multiple implementations of test data providers. A random test data generator can be one of the implementations and other could be implemented as required. This will ensure that test logic is not tightly bound with test data and can change independently of test data.

 d. Change test data.

If you invest in a test data strategy factory, this will be an added advantage since there could be a test data strategy to modify data for each test run or for any run based on some rules relevant to the scenario at hand.

There are multiple programming techniques which could aid in effective test data planning and execution. Important thing to remember is that, test data should be given equal importance if not more, as actual test logic. This approach will yield multiple benefits in future in terms of re-usable test data which makes tests more useful & effective with each run.

To be continued…