Ruby: The misconceptions of 100% Code Coverage
There are misconceptions around the value of having 100% code coverage, some believe it guarantees them bug free code, others see it as a waste of time. With Ruby being a dynamic language I believe code coverage plays an important role.
Code coverage is a statistic of how many lines of code are executed during the running of a test suite out of the total number of lines in the application, it is a percentage score. If I run a test suite and the ruby interpreter doesn’t go down one of the conditional branches within a rails controller action then the percentage is going to be below 100%.
Often, 100% code coverage is touted around as if it guarantees the code being bug free and robust, but in reality it only protects you from a handful of runtime exceptions. Your tests can be passing, but in production there maybe state or data in use that your tests do not account for, so even though the production code is covered by a test it doesn’t mean that it will not break.
Ruby does not have a compiler and is missing out on basic checks that other languages take for granted. When you execute a ruby file there are a few things you can do that will cause the ruby interpreter to fail and exit the process, but the majority of exceptions encountered in ruby development are runtime exceptions. Ensuring code coverage does a small part of the job of a compiler, it ensures that your code can at least be run!
The basic exceptions that exercising a line of code can catch include MethodMissing, ArgumentError, NameError, and many others like Hash#KeyMissing from standard library related errors to integration with third party gems.
Imagine that you just finished implementing a feature but you haven’t pushed up your code for review just yet because you decide to rename one of the methods you are using, your tests pass, it passes code review and you deploy the feature. A production bug comes back to you later in the day that one of the pages is broken for a few users only, so you investigate and find that a MethodMissing exception is being raised. You scratch your head because it is the original method name that you had renamed, but then you spot that it is being used in a different place! You had missed a small code path that covered an edge case, for a subset of users, but it did not have any tests for it. Bugs like these are too easy to get into production.
Having 100% code coverage would have caught that this other method needed renaming too. Of course an even better scenario would have been that you had been using Test Driven Development to implement that edge case so that you would have had tests covering that code by definition, but sometimes it can be tempting to plug a small edge case during TDD’ing a feature without writing a test for it. If you configure your CI pipeline to fail if 100% code coverage is not hit then you will be given the opportunity to add proper testing for these scenarios.
Without these basic checks, changing the arity of a method, renaming a local variable, and many other refactorings become risky that would benefit the code for future developers, or yourself in several months time. I am a serial typo developer and prone to raising basic runtime exceptions.
After just a few weeks into a new project the code can become too big to fit into a single developers mind, so if you are leaving gaps in your test coverage then you are leaving a ticking time bomb. As a new developer on a mature project I do not want to be in constant fear that my ignorance of large parts of the application and a lack of a warning system is going to break production.
For new projects I recommend starting with 100% code coverage as mandatory, which is simple to add in using SimpleCov and configuring it to fail the build below a threshold. For existing projects because of the sheer workload of having to write the tests to cover code that you have not written to fill the gaps, it is much harder to get to the confidence level that having 100% coverage brings, but at least by using code coverage reports you can make informed decisions on how risky a change to parts of your application are going to be.
For projects that are lacking in code coverage I would recommend a third party tool like Coveralls, CodeCov or CodeClimate to have tracking in how your code coverage is changing over time and a good interface to dive into to see where coverage has been missed and how severely.
For small Rails projects, after running the test suite locally with SimpleCov enabled then you can open coverage/index.html in your project root directory to see a rundown of your application code’s covered and missed lines.
Remember, the confidence of 100% code coverage is not that your code is bug free, it’s that is has passed the most basic test: that your code can be executed and therefore you are avoiding a slurry of runtime errors that can too easily make it into production. A preventative is worth ten times the cure.
The effects of having 100% code coverage VS less cannot be tangibly measured, but if I were joining a project that is even only a year old I would feel much happier knowing that I can perform simple refactorings safely. If you have been around in the Rails industry for any length of time it is likely that you have come into contact with projects that are 5 or even 10+ years old and the fear of breaking things, imagine if they had started with 100% code coverage - would they be a nicer project to work on? I like to think that they would: the developers would feel more confident in their work and that the project is overall better off for it.
These thoughts, ideals and practices leave me wanting to do right for the people who work on my projects years down the line. Even if that is me.