Writing Tests
/Introduction
Writing tests is easy. We just fire up our favorite IDE, create a JUnit test case, write a bunch of test setups, executions, assertions, and it is done.
Fast forward a few months later, and we have to make a change that would break an existing test. We make the change, and fire up all related tests.
Red bar.
So we fire up the IDE, and look at the test case that failed.
"What is it trying to set up? What is this executing? What are all those assertons?"
We give up, find the assertion that failed, make random changes in both the tests and the class, and rinse and repeat until the test pass. And we go on with our merry lives.
...
But it does not have to be this way.
What is the Problem?
So what went wrong with our development approach?
We employed Test-Driven-Development, where we write tests first. We have 100% test coverages.
But these are just lip services we pay. No matter how much of these principles we apply, if we do not have a sound foundation, all these hurt more than help us.
Purpose of Article
This article hope to describe the approaches and methods learnt from various sources of writing tests, with the eventual goal of tests that are easy to understand and maintain.
General Template - 'Given, When, Then, Cleanup' Layout
I first learnt of this 'template' from my BDD(Behavior/Specification Driven Development) adventures.
When we write JUnit tests, we usually start off with a @Before, then a test method, and an @After. Within the test method, we might do additional test setup, then invoking a target method, and then asserting that the post-conditions are correct.
If we categorize the above actions, we would have easily end up with the following categories of operations:
Given - @Before and additional test setup in test methods.
When - Target method invoked for which this test is written for.
Then - Assertions that the post-conditions are correct, or expecting an exception.
Cleanup - Cleaning up the resources used in testing, eg. Database connections.
Let's review a typical JUnit test case first.
public class MyTest {
// test variables
@Before
public void setup() {
// setup 1
}
@Test(expected=MyException.class)
public void expectException() {
// setup 2
// invoke method to test
}
@Test
public void success() {
// setup 3
// invoke method to test
// assertions
}
@After
public void cleanup() {
// cleanup
}
}
In this example, for a given test, you had to lookup two places to know what is the test setup - the 'setup 1' code block of the @Before method, and 'setup 2/3' code block in the test method. One cannot, at a single glance, know what a test method is 'given'.
A quick refactor yields this:
public class MyTest {
// test variables
@Test(expected=MyException.class)
public void expectException() {
// given
// setup 1
// setup 2
// invoke method to test
}
@Test
public void success() {
// given
// setup 1
// setup 3
// invoke method to test
// assertions
}
@After
public void cleanup() {
// cleanup
}
}
Of course, after this, the cleanup seems weird to be standing along by itself. Could be better if they are close to the 'given' block (and the test variables need no longer be class instances):
public class MyTest {
@Test(expected=MyException.class)
public void expectException() {
// test variables
try {
// given
// setup 1
// setup 2
// invoke method to test
} finally {
// cleanup
}
}
@Test
public void success() {
// test variables
try {
// given
// setup 1
// setup 3
// invoke method to test
// assertions
} finally {
// cleanup
}
}
}
The invokation of methods would correspond directly to our 'when' block, so there is no problem there.
Let us look at the 'then' of the method invokation. One of them expects an exception, while the other performs assertions. We would have to look at two places to find out what really happens after a method is invoked! This is just not consistent enough. Two places are few, but what we really want is just a fluent flow of 'given - when - then'.
This layout would fit more naturally to what we wanted:
public class MyTest {
@Test(expected=MyException.class)
public void expectException() {
// test variables
try {
// given
// setup 1
// setup 2
try {
// invoke method to test
} catch(MyException t) {
// ok!
}
} finally {
// cleanup
}
}
@Test
public void success() {
// test variables
try {
// given
// setup 1
// setup 3
// invoke method to test
// assertions
} finally {
// cleanup
}
}
}
In fact, we could generalize the above template as a method, to be used for both test cases where you expect exceptions in one, and none in the other:
public void test() {
// test variables
try {
// given
Throwable ex = null;
try {
// when
} catch(Throwable t) {
ex = t;
}
// then(ex)
} finally {
// cleanup
}
}
If we are in a language that supports closures, we could simply plug this template method with given, when, then, cleanup code blocks, an extremely powerful and consistent approach to writing our testing. But given that I am working in a purely Java environment, I can only manage to come up with the following:
public abstract class TestScenario {
public void run() {
try {
try {
given();
Throwable e = null;
try
catch(Throwable e2) {
e = e2;
}
then(e);
} finally
} catch(TestFailedException e) {
throw e;
} catch(Throwable e) {
throw new TestFailedException(e);
}
}
protected void given() throws Throwable {}
protected void when() throws Throwable {}
protected void then(Throwable e) throws Throwable {}
protected void cleanup() throws Throwable
And when I need to write a test, I do the following:
@Test
public void doTest() {
new TestScenario() {
// test instances
@Override
protected void given() {
// setup test environment
}
@Override
protected void when() {
// execute test method
}
@Override
protected void then(Throwable e) {
// assert there is an exception or not
// assert other post-conditions
}
}.run();
}
Extract Code Blocks From Tests
Whenever a code block in a section grows too much, readablity suffers. At this point, we should really refactor the code and extract them out as methods.
Instead of the following:
protected void given() {
// delete old file
// open stream to new file
// write data to new file
// flush stream
// close stream
}
We could have:
protected void given()
void prepareTestFile() {
// delete old file
// open stream to new file
// write data to new file
// flush stream
// close stream
}
The benefit of this is clear. We get easier to read code block, and we get to reuse the setup code.
Scenario-Based Test Setup
Many times, for a given scenario, many actions can be performed, with many different input. In the good old JUnit days, these would be done in the @Before method, and the various actions and input are placed in a method of its own.
But as mentioned before, we really would like to know, at a single glance of only the 'given' block, what was the environment like. Therefore we rely on the given block to set up the environment for us.
Of course, with so many tests, the setup and cleanup code would duplicate. To solve this, we introduce Scenario-Based Test Setup classes, which house the scenario code. The tests would all use the scenario classes to perform setup and cleanup. Example:
class AppDbSetup {
void prepareTable()
void populateTestData()
...
protected void given() {
AppDbSetup.prepareTable();
AppDbSetup.populateTestData();
}
Do Proper Cleanup
One important thing to remember is that each test should do its own cleanup. If the test created test files on the file system, or added test data to the database, it should remove them after the test execution. This is to prevent the previous tests from affecting any latter tests.
Taken this to the other side of the coin, each test should make sure the environment is set up properly too. If it did not expect a file on the filesystem, it should delete it thoroughly!
Assertion Checks
During the 'then' phase of the test, we would be performing assertions check to verify the post-condition of the system state after the 'when' execution.
However, at many times, we are overwhelmed by the excessive checks we have. We can reduce this clutter though, by removing unnecessary checks. For example, with the following lines:
assertNotNull(result);
assertEquals(5, result);
The first assertion is unnecessary, and can be removed.
Making Assertion Checks more Readable
The best way to make assertions more readable are to hide all of it! Consider the following assertions:
int[] results = ...;
int[] expected = ...;
void then(Throwable e){
assertNull(e);
assertEquals(expected.length, results.length);
for(int i = 0; i < expected.length; ++i) {
assertEquals(expected[i], results[i]);
}
}
The above block could be better served with the following method calls instead:
void then(Throwable e){
noException(e);
arraysAreEqual(expected, results);
}
Of course, the code for the assertion checks are still as long, and we introduced more methods into the test class. But the key concern we have here is better readablity on the test case. If additional details are required, we can go check out what those methods do. But in more cases, such a high level overview is sufficient.
Personally, I took a step extra and wrote a more fluent interface for my assertions check. The resulting assertions look as follows (for simplier case):
check(ex).isInstanceOf(IllegalArgumentException.class);
check(file).isNull();
check(result).isTrue();
check(dir).isFalse("exists");
And the more complex ones:
check(instance).get("systemBundleContext").get("bundles").length().isEquals(1);
check(instance).get("systemBundleContext").exception().isInstanceOf(InstanceNotStartedException.class);
The extra time spent in writing more readable test code would pay itself off in the future when we come back to maintain them.
Injecting Dependencies for Easier Testing
Probably one of the more important aspect of testing is to make sure the target class is easy to test. Extract possible environment dependencies out, and make sure that it is easy to set up the possible test environments. An example is to extract database connectivity out as a class, and inject it into the tested class:
class IDb {
void insert(Object o);
Object get(int id);
}
class TestDb implements IDb {
void insert(Object o)
Object get(int id)
class BusinessObject {
private IDb db = ...;
void setDb(IDb db)
void useDb() {... // use db
}
}
This allows you to do isolated unit testing without wiring up to the database first, making testing fast and cheap. Of course, this is not restricted to environmental dependencies. Business logic could be injected as well if it aids in testing.
Mocking Injected Dependencies
In the previous example, we created a TestDb to aid in testing. Practically, however, for each test case, TestDb might perform differently. Creating a subclass of IDb for each test case is simply overkill. And so we use Mocks.
Mocks allow us to simulate an implementation of an interface for testing purposes. With mocks, we eliminate the need for multiple subclasses, and we can perform verification on the mocks themselves. To better explain, an example illustration would be beneficial:
db = mock(IDb.class);
// inject db into BusinessObject
// use BusinessObject
// verify mock calls
verify(db).insert(o);
The above example uses mockito, and the last call checks to see if db.insert has been used between its creation and the verification. The call also check that the right argument is passed in.
Mocks are actually used more commonly in white-box testing, as we have to be more aware of how the mocks are used in the injected objects.
Although the example shows verification of mocks, they can also be used in situations of method stubs, where you wanted a method of the mock to return a specific value when invoked.
Conclusion
This is by far not a completed article, nor the only way to do tests. This article merely details what employing now, and what works for me. Comments and suggestions are freely welcomed, to help to evolve my approach into writing better tests, and improving this article.