08 January 2015

Python test coverage tools

I guess there are many metrics out there to know how effective are your unittest proofs to cover all possible cases of your developments. In the future I'm going to formalize my knowledge in TDD but nowadays I'm just playing with the concepts so I follow a very simple metric: If my tests ejecutes all my code program then I'm doing OK.

How to know if your tests executes all your code?. When code grows so your tests amount do, then it's easy to miss a fragment of code and leave it untested. There is where test coverage tools come to help you.

Those tools follow your tests while they execute your code and take note of visited code lines. That way, after tests execution you can see statistics about which percentage of your code is actually tested and which not.

We are going to see two ways to analyze your test coverage: from console and from your IDE.

If your are in an Ubuntu box you should install "python3-coverage" to get coverage analysis in the console:
dante@Camelot:~/project-directory$ sudo aptitude install python3-coverage

Once installed, python3-coverage has to be called to run your unitests:

dante@Camelot:~/project-directory$ python3-coverage run --omit "/usr/*" ./run_tests.py

I use "--omit" flag to keep "/usr" out of coverage reports. Before using that flag calls of my program to external libraries were included in the report. As I don't test external libraries because they are not developed by me, getting their coverage statistics would make my reports harder to read. The script "run_tests.py" is the same I explained in my article about unittest.

Suppose all your test run correctly, you can generate a report about your coverage in html or xml format. To get a report in html format, just run:

dante@Camelot:~/project-directory$ python3-coverage html

This command generates a folder called "htmlcov" where your html report is stored (index.html is its entry page). The only thing a bit a annoying is that htmlcov has to be removed manually before generating a new one. Besides it's boring to search generated index page to open report. That's why I prefer to use an script to automate all those boring things:

#!/usr/bin/env bash

python3-coverage run --omit "/usr/*" ./run_tests.py
echo "Removing previous reports..."
rm -rf htmlcov/*
rmdir htmlcov
echo "Removed."
echo "Building coverage report..."
python3-coverage html
echo "Coverage report built."
echo "Let's show report."
(firefox htmlcov/index.html &> /dev/null &)
echo "Bye!"

Running that script (I call it: "run_test_and_coverage.sh") I get an automatically opened firefox browser showing your just created coverage report.

If you use and IDE, chances are that it includes some sort of coverage tool. PyCharm includes coverage support in its professional version. Actually with PyCharm you get more or less the same than from console tools but integrated with your editor in a more confortable way.

At application level, default configuration should be enough:

I guess if you have not installed system package "python3-coverage" you should check "Use bundled coverage.py" option to use native coverage tool included with  PyCharm. In my case I haven't notice any difference either checking or unchecking that option (obviously I have "python3-coverage" installed).

The only tricky thing is to remember that running test with coverage support has its own button in PyCharm interface. That button is located next to "Run" and "Debug" ones:

After using that button, you get a summary panel right your main editor with percentages showing coverages in each folder. Outliner at left hand side marks coverage by each source file. Besides, your main editor window will get colored to mark lines covered with green color and not covered in red.

Click image to enlarge

Keep your test coverage as near as possible to 100% is one of your best indicator your tests are well designed. To control it you can use console or your IDE tool, its your choice, but both of then are easy enough to use them often.