Sufficient Tests

I’ll allow up front that I am not a huge advocate of TDD. Not because I think it is bad, its good. Not because I think it is hard, although it adds abstractions to the development process that are hard for some developers to grok. Not because I think it is a waste, because even though it adds time up front, it can save double on the back end. I am not a huge advocate of TDD, simply because it has the developers writing the tests.

Over years and years of software delivery experience, as a developer, as a tester, as a project manager, a business analyst, a manager I have observed one truism. Software developers cannot be trusted to understand the requirements deeply enough to test their own code. There are too many layers of abstraction in the way.

As Seymour Cray is reported to have said,

So let me break down the practices within TDD to explain why cumulatively, they are insufficient to ensure correct operation in the initial case.

1) Level of Specificity

Requirements are often written at a level that implicates many layers and components within the software system. Requirements are written ex of implementation. Unit tests are written against the implementation, and when done correctly, encapsulate the operational integrity of a single component. This requires the developer to “see” across the layers, to construct the “fleet” of unit tests that prove that a requirement is met.

2) Semantic Confusion

Developers often rely on the names of classes, methods, functions or properties to know what they mean. Sometimes unfortunate incidents of naming create semantic confusion, and causes the developer to use the wrong property, or method as an input to another method or function. The class or method the developer made has an appropriate unit test mocking the input, but this cannot detect that it is not the correct input.

3) Structural Insufficiency

Developers often rely on requirements to specify valid application data state. The truth is that much of the validation is dependent on the structure as designed by the developers. Nobody but the developers would know that the semantics are dependent on structure, that the design of the class model implies some states that the requirements do not specify as explicitly invalid, are structurally invalid within the application. This gives way to the traditional “null reference” types of issues, where some variable/object is not instantiated in every case when it is referenced. These technical requirements are only known by the developers, and are completely dependent on the structure of the software. Sometimes it is as simple as the order of execution – I ask for something before I have created it.

Here is a general case where TDD can break down:

1) Developer 1 (Bob) creates a class to represent accounts. For the requirements that he is aware of, he creates unit tests covering the public interface of the account class.
2) Developer 2 (Fred) needs properties of the account class as an input to his account grouping class. Some of those properties default to null when the object is instantiated, and certain operations must be performed in order for them to be valid for his needs.
3) Bob’s unit tests align with the assumption that it is valid for those properties to be null until they appropriate operations are performed.
4) Fred’s unit tests mock the account class properties as valid, because he is unaware that null is a valid state.
5) Fred’s class is currently behind a User Interface can cannot be navigated to until the appropriate operations are performed on the account.
— In the current application context, there is no observable problem, because the requirement for the operational integrity is maintained by logic in the user interface separate from the model classes.
6) In a subsequent release, after Fred and Bob roll off the project, developer 3 (Shelby) builds a new user interface that relies on Fred’s account grouping class that is not restricted for navigation until the operations are performed.
— Now, the problem becomes observable because there is nothing preventing the request from being made when the account class is in a state that does not support the requirement of the client class.

What to do?

1) Understand User Experience – Every developer should be able to run test cases through the user interface. Every developer should be able to trace information from the display or output back to persistence. Every developer should be able to work happy path cases through user interfaces that they are working with or working behind.
2) Understand the non-technical domain – Nothing, in my mind, is a substitute for developers understanding the user domain of the application. Developers relying on analysts or testers to verify their understanding of a particular class or behavior or structure is a poor substitute for actually grokking the user’s semantic domain and assimilating the knowledge domain of the customer.
3) Code Proper English – In my experience, most of the semantic problems in an application are caused by developers who don’t understand the business case. They create names that don’t align with the non-technical domain. These names in turn make it harder for everyone else to understand the logic as implemented. If you name things so that you can read your code in reasonable english sentences – it will be easier for everyone.
4) Understand In-use Technical Abstractions – If you are working on an application that is using a layer design pattern like MVC or MVP or MVVM or some other high level abstraction, work to understand what implementation practices are considered good within that technical abstraction. Don’t recommend abstractions that you don’t already know how to implement – Don’t be afraid to say you don’t know how to correctly implement the in-use abstractions. Your mates would rather teach you how to do it right than replace your code after you’ve been fired. Learn from them – because they sure don’t want to do both.
5) Understand Design Principles – Code design is less important than layer design. Layer design is less important than domain modeling. Spending time designing the model around which the application is built, and reviewing every change to model classes to ensure that the model is semantically correct is key to maintaining the simplicity of an application. When your model classes are not semantically correct, you usually find lots of code in services and UI working around the core problem that is semantic.
6) Understand Responsibility – TDD is useless when we make poor decisions about which class/layer/service is responsible for what behaviors. Some of this is semantic – how you describe a behavior will influence where you assign responsibility.
7) Understand Testability – Some code is easier to write tests against than others. If you have complex logic (composed of simple control structures) or longer sequences inside of loops in a method – you may have to write many tests to prove that all “paths” through the loop work correctly. Invert your thinking – declare each “path” as a method, and create a path “decider” method or property that returns a path and feed that into a case statement that calls each path method. Then, if some of your paths share common code, factor that out as separate methods. Not only is this code design pattern easier to write tests against, it is easier to assign responsibility across classes and layers.

—
These do’s are mostly oriented to the object oriented paradigm – there are a similar set of do’s for other paradigms. Functional and procedural paradigms have similar rules.