Writing unit tests is more of an art than a skill, and understanding what to test for comes with experience and/or mistakes. We look for the percentage of test coverage for examining the health of an application. However, the "coverage percentage" might be misused or overlooked. In particular, it falls victim to Goodhart’s law, which says: “When a measure becomes a target, it ceases to be a good measure”.
Whenever we start writing test cases for the sake of improving the code coverage, we miss the whole point of testing and rather introduce Tautological Tests - poorly designed tests written with the sole intention of meeting coverage requirements instead of testing the unit, feature and overall software correctly.
In this article, we will examine what is Tautological Test and how to avoid them in a Ruby on Rails application.
Tautology in general?
Needless repetition of an idea, statement, or word.
Tautology in coding?
It refers to a testing pattern where expected and actual parts are equal in some respect such that it makes them useless or meaningless.
Tautology in a comic-strip?
Tautological tests problems:
- Developer gets a false sense of satisfaction about his/her "well-tested" code with tautological tests. High code coverage percentage generates a false sense of confidence in developers. And they get very comfortable in pushing changes as long as the tests pass, even though those tests aren't testing the actual business functionality or behaviour.
- Tautological tests check the implementation rather than testing the behaviour of code. One needs to update the tests, whenever there are any changes in implementation rather than changing tests when the expected output changes. It motivates engineers to update tests instead of probing for the reason of failure.
Examples of Tautology tests in Rails context
- Testing the presence of associations
it { expect(author).to have_many(:articles) }
The sole purpose of this test is to verify if association has_many: articles is present on Author model. I Agree that the test is not incorrect and adding it will do no harm. But it feels like we're testing whether we have written the associations properly than whether the associations work properly.
Concession: There are instances where associations get complicated with conditions, dependencies, class names etc. In this situation, a direct test of association existence comes in handy rather than implicitly covering each part in other behaviours.
- Testing that an object responds to certain methods
it { expect(author).to respond_to(:full_address) }
Above test verifies that the Author instance responds to full_address method. A better approach here can be to test the method's behaviour.
Concession: There are different ways by which we can make the method available to class such as using includes, aliases or inheritance. expect(x).to respond_to(:method_name) is a straightforward way to verify that the method is present.
- Testing for the presence of indexes
it { expect(article).to have_db_index(:account_name)
Above test verifies that uniqueness index is present for account_name on Article model. A finer approach can be to validate that we get an error message if we try to create a duplicate.
Concession: During TDD it comes in handy. Writing tests with has_db_index helps developers to follow red, green, refactor pattern. So it's not incorrect to write such tests during the TDD approach. But as the application expands and actual scenarios are covered, it's more fruitful to write the duplication test for validation.
- Testing for the presence of callbacks
it { expect(article).to callback(:create_summary).before(:save) }
Above test verifies that if create_summary method is fired before saving the article object. But a better approach here can be to test the behaviour of create_summary.