A simple, effective test automation strategy

DevOps teams automate everything. When you’re getting started with test automation, it’s important to wonder why someone would want to automate tests. And what is testing, anyway? This article describes what testing is and what parts of that process can be automated. Fortunately, there’s no way test automation will eliminate manual testing; it will just make that process more efficient.

Here’s why:

It’s not the client’s idea that goes to production, it’s the developer’s assumption that does..

Communication begins with an idea. The brain converts that idea into words, and the mouth transmits those words through the air. The listener’s ear receives them, and the brain interprets the words and tries to create the same picture the transmitter had. However, not surprisingly, the most significant part of the initial message gets lost in this process.

Since most of the initial message gets lost in translation, a big part of testing is about communication; talking to the client and the developer. Are there any misinterpreted, overlooked, or missing requirements? Testing is about finding the difference between the client’s initial idea and the developer’s interpretation of it.

You can’t automate testing

Don’t confuse testing with checking. Checks can be automated, testing can not. When you’re testing, you’re getting to know the (new) behavior of the system. It takes human intelligence and interaction to judge whether the behavior is right or wrong — that can not be automated.

Generally speaking, when software changes (new features are implemented, code gets refactored, etc.), you want to make sure that the system’s existing behavior hasn’t changed. To do this, you must validate the software’s features by running specific (combinations of) parameters through the system. Define the correct outcome upfront, so it can be easily validated. This process is also known as checking, which can be automated.

Automate checks to test more efficiently

In a continuous-delivery world, software gets shipped all the time. That means somebody needs to check the known features and to test the new ones thoroughly. Automating the recurring work results in fewer errors and more time to do manual exploratory testing properly.

Normally, untested software doesn’t get shipped. The shorter it takes to validate the software, the faster and more often the product can be shipped. Specify the desired state of the system in a set of quick, automated checks that can run any time, so you can test and ship the system any time you want.

Don’t be mistaken. Running automated checks against existing features indicates that the system still does what it is supposed to do. Nothing less. But what if it turns out that the system does more than was specified? Is that good or bad? Test existing functionalities occasionally to find bugs and append automated checks.

Implement Martin Fowler’s test pyramid

Apply different types of testing for the problem at hand. Don’t unit test everything. Make sure the resulting test-suite runs quickly and pinpoints the location of the problem in the code so issues can be found and fixed fast. Use as many of the cheapest and fastest automated checks as you can and slow, expensive checks carefully.

the testing pyramid

Think of your test-suite like a pyramid, with the base made of many unit tests. They’re cheap and fast, and you need lots of them. The developers own these and should use test-driven development to produce such a great amount of tests. The middle of the pyramid should have a substantial amount of tests that validate whether the individual units together produce the expected business value. Use behavioral-driven development for this layer.

The component/integration layer tests the functional aspect of the application, and the unit tests test the edge-cases of the application. The top layer is the slowest. Use this layer to test the deployed application — ensure the components rendered and the app can launch. Use only a handful of these tests because they are slow and costly.

You can not apply this strategy to any codebase; it requires SOLID principles.

Use Test Driven Development to speed up the coding process

If you start automating checks, start with Test Driven Development. Beware! Test Driven Development is not about testing. It’s a software development method. It helps designing proper software and speeds up the development process.

For maintenance reasons, code should be spread out over multiple files. They all have one single responsibility and few ifs. That means those need to be put together to be able to see if all pieces put together, work. Test Driven Development allows the developers to get feedback faster. Each component can be tested individually, allowing a much shorter feedback circle.

Unit tests don’t often validate business value. Units are usually too small and technical to allow testing business value on that level and their outcome can be manipulated elsewhere in the application. Assuming that unit tests always validating business value can create false positives. Beware of that! Validate the essential business value in the component/integration check layer instead.

Use a proper naming convention that describes behavior and makes it obvious why a test needs fixing, and what needs fixing when it fails. Something like: WhenTooManyProductsSelected_ShouldDisableAddToBasketButton().

Use Behavior Driven Development to create living documentation

It doesn’t make sense to test software that hasn’t changed — unless it’s properly documented. Testing documents the system so developers can change it. Automated checks are the documentation of the system and they should make changing the system easier not harder.

How trustworthy is a test result if classes change and their tests have been refactored as well? Unit tests are tightly coupled to the class structure. As a result, they’ll change often. Some will be removed, some will be changed, and some will be added.

Put several units together to test business value on a component integration level. You don’t have to change the tests that contain the business value all the time, which makes the results more trustworthy and gives the developers the freedom to refactor the code whenever they want.

Use the Gherkin language to define your specifications so your client can validate that you’re testing the right things. Link the specification to code using Cucumber, SpecFlow or some other BDD framework to create living documentation that’s always up to date.

Make sure the application runs as expected

After completing the component/integration tests of all the functional requirements and unit tests of all the edge cases, all that remains to test are the infrastructure, configuration, and permissions.

Does the application start and connect to the database? Has it been configured properly? Automate these tests, too, and test them on a deployed system.

Make sure all components are used in the expected places and use BDD to specify these places. It’s important that this information is in the system’s documentation.

Test the test!

All code needs testing, including automated checks. These are meant to fail and warn the team when code changes and causes the system to break. Test these by changing if-statements in the code then run the tests to see if the tests fail. You’ve got work to do when they don’t. This process is called “mutation testing” and can be done manually or by using frameworks.

Also, keep monitoring! I can’t tell you how often I’ve seen passing tests and a broken production environment. Checking the log files for errors doesn’t cut it. Who says errors are being logged? Make sure conversion points are still being reached. Use the feedback provided by the production environment to complete the test cases.

Summary

Test automation doesn’t exist. Check automation does. Use automated checks to test more efficiently and to make sure the state of the system can always be quickly determined. Implement Martin Fowler’s test pyramid and use TDD to speed up the coding process. Test the requirements of the system on a more abstract level to allow easy refactoring. Use only a few technical tests to validate infrastructure and permissions. Don’t blindly trust your automated checks, test the test, too.