Simple unit testing with a Makefile

Automated unit tests are very useful. They’re an excellent way of making sure you haven’t broken something in an obvious way when you change things. You can also implement them with very little work and without needing to pull in an external framework, just by using Makefiles. Since Make understands dependencies, this also ensures that when edits are made, only the minimal number of tests need to be rerun.

This is slightly simplified example on the method I’ve been using on the TooN library and my own projects.

Let’s say you have a very simple matrix class:

#ifndef MATRIX_H
#define MATRIX_H

#include <cmath>
#include <initializer_list>
#include <array>
#include <cassert>
#include <iostream>
struct Matrix2
{
	
	std::array<std::array<double, 2>, 2> data;

	public:

		Matrix2(std::initializer_list<double> i)
		{
			assert(i.size() == 4);
			auto d = i.begin();
			data ={ *(d+0), *(d+1), *(d+2), *(d+3)};
		}

		std::array<double,2>& operator[](int i)
		{
			return data[i];
		}	

		const std::array<double,2>& operator[](int i) const
		{
			return data[i];
		}	


		Matrix2 operator*(const Matrix2& m) const
		{
			Matrix2 ret = {0,0,0,0};

			for(int r=0; r < 2; r++)	
				for(int c=0; c < 2; c++)	
					for(int i=0; i < 2; i++)
						ret[r][c] += (*this)[r][i] * m[i][c];
			return ret;
		}

};

inline double norm_fro(const Matrix2& m)
{
	double f=0;
	for(int r=0; r < 2; r++)	
		for(int c=0; c < 2; c++)	
			f+=m[r][c];

	return sqrt(f);
}

inline Matrix2 inv(const Matrix2& m)
{
	double d = 1./(m[0][0]*m[1][1] - m[1][0]*m[0][1]);

	return {
		 m[1][1]*d,   m[0][1]*d ,
		 -m[1][0]*d,  m[0][0]*d 
	};
}

std::ostream& operator<<(std::ostream& o, const Matrix2& m)
{
	o<< m[0][0] << " " << m[0][1] << std::endl;
	o<< m[1][0] << " " << m[1][1] << std::endl;
	return o;
}

#endif

(did you spot the error?)

And you want to find the inverse of a 2×2 matrix:

#include "matrix.h"

using namespace std;

int main()
{
	Matrix2 m = {1, 1, 0, 1};
	cout << "Hello, this is a matrix:\n" << m << endl 
	     << "and this is its inverse:\n" << inv(m) << endl;

}

Simple enough. In order to build it, you can write a very simple makefile:

CXX=g++-5
CXXFLAGS=-std=c++14 -g -ggdb -Wall -Wextra -O3  -Wodr -flto


prog:prog.o
	$(CXX) -o prog prog.o $(LDFLAGS)	

We can make:

make

And get a result:

Hello, this is a matrix:
1 1
0 1

and this is its inverse:
1 -1
-0 1

Plausible, but is it right? (a clue: no.) So, let’s write a test program that creates matrices and multiplies them by their inverse and checks their norm against the norm of I. This will go in tests/inverse.cc


#include "matrix.h"
#include <random>
#include <iostream>
using namespace std;


int main()
{
	mt19937 rng;
	uniform_real_distribution<> r(-1, 1);
	int N=10000;
	double sum=0;
	
	for(int i=0; i < N; i++)
	{
		Matrix2 m = {r(rng), r(rng), r(rng), r(rng) };
		sum += norm_fro(m * inv(m))-sqrt(2);
	}

	cout << sum / N << endl;

	//Looks odd? Imagine if sum is NaN
	if(!(sum / N < 1e-8 ))
	{
		return EXIT_FAILURE;
	}

	cout << "OK\n";
}

And we get the output:

6.52107

So there’s an error. At the moment the test is ad-hoc. We have to remember to compile it (there’s no rule for that) and we have to remember to run it whenever we make some edits. This can all be automated with Make.

So, let’s first make a rule for building tests:


#Build a test executable from a test program. On compile error,
#create an executable which declares the error.
tests/%.test: tests/%.cc
	$(CXX) $(CXXFLAGS) $< -o $@ -I . $(LDFLAGS) ||\ { \ echo "echo 'Compile error!'; return 126" > $@ ; \
	  chmod +x $@; \
	}

This is a bit unusual, instead of just building the executable, if it fails, we make a working executable which indicates a compile error. This will eventually allow us to run a battery of tests and get a neat report of any failures and compile errors rather than the usual spew of compiler error messages.

So now we can (manually) initiate make and run the test. Note that if the test fails, the program returns an error.

We’re now going to take this a bit further. From the test program we’re going to generate a file with a similar name, but that has one line in it. The line will consist of the test name, followed by a status of the result. We do this in two stages. First, run the test and append either “OK”, “Failed” or “Crash!!” to the output depending on the exit status. If a program dies because of a signal, the exit status is 128+signal number, so a segfault has exit status 139. From the intermediate file, we’ll then create the result file with the one line in it.

#Build a test executable from a test program. On compile error,
#create an executable which declares the error.
tests/%.test: tests/%.cc
	$(CXX) $(CXXFLAGS) $< -o $@ -I . $(LDFLAGS) ||\ { \ echo "echo 'Compile error!'" > $@ ; \
	  chmod +x $@; \
	}

#Run the program and either use it's output (it should just say OK)
#or a failure message
tests/%.result_: tests/%.test
	$< > $@ ; \
	a=$$? ;\
	if [ $$a != 0 ]; \
	then \
	   if [ $$a -ge 128 ] ; \
	   then \
	       echo Crash!! > $@ ; \
	   elif [ $$a -ne 126 ] ;\
	   then \
	       echo Failed > $@ ; \
	   fi;\
	else\
	    echo OK >> $@;\
	fi


tests/%.result: tests/%.result_
	echo $*: `tail -1 $<` > $@

We can now make test/inverse.result and we get the following text:

g++-5 -std=c++14 -g -ggdb -Wall -Wextra -O3  -Wodr -flto tests/inverse.cc -o tests/inverse.test -I .  ||\
        { \
          echo "echo 'Compile error!'" > tests/inverse.test ; \
          chmod +x tests/inverse.test; \
        }
tests/inverse.test > tests/inverse.result_ ; \
        a=$? ;\
        if [ $a != 0 ]; \
        then \
           if [ $a -ge 128 ] ; \
           then \
               echo Crash!! > tests/inverse.result_ ; \
           else\
               echo Failed > tests/inverse.result_ ; \
           fi;\
        else\
            echo OK >> tests/inverse.result_;\
        fi
echo inverse: `tail -1 tests/inverse.result_` > tests/inverse.result

And the contents is:

inverse: Failed

Just to check the other options, we can add the following line to tests/inverse.cc

*(int) 0 = 1;

And sure enough we get:

inverse: Crash!!

So it seems to be working. The next thing is to be able to run all the tests at once and generate a report. So we’ll add the following lines to use every .cc file in tests/ as a test and process the strings accordingly:

#Every .cc file in the tests directory is a test
TESTS=$(notdir $(basename $(wildcard tests/*.cc)))


#Get the intermediate file names from the list of tests.
TEST_RESULT=$(TESTS:%=tests/%.result)


# Don't delete the intermediate files, since these can take a
# long time to regenerate
.PRECIOUS: tests/%.result_ tests/%.test


#Add the rule "test" so make test works. It's not a real file, so
#mark it as phony
.PHONY: test
test:tests/results


#We don't want this file hanging around on failure since we 
#want the build depend on it. If we leave it behing then typing make
#twice in a row will suceed, since make will find the file and not try
#to rebuild it.
.DELETE_ON_ERROR: tests/results 

tests/results:$(TEST_RESULT)
	cat $(TEST_RESULT) > tests/results
	@echo -------------- Test Results ---------------
	@cat tests/results
	@echo -------------------------------------------
	@ ! grep -qv OK tests/results 

Now type “make test” and you’ll get the following output:

-------------- Test Results ---------------
inverse: OK
-------------------------------------------

The system is pretty much working. You can now very easily add tests. Create a .cc file in the tests directory and make it return s standard code and… that’s it. The very final stage is to make the target we want to build depend on the results of the test:

prog:prog.o tests/results
	$(CXX) -o prog prog.o $(LDFLAGS)	

At this point you can now type “make prog” and the executable will only build if all the tests pass. There’s one minor wrinkle remaining: make has no mechanism for scanning C++ source files to check for dependencies. So, if you update matrix.h then it won’t rerun the tests because it doesn’t know about the dependency of the test results on matrix.h. This problem can also be solved in make. The complete makefile (with the dependency scanner at the bottom is):

CXX=g++-5
CXXFLAGS=-std=c++14 -g -ggdb -Wall -Wextra -O3  -Wodr -flto


prog:prog.o tests/results
	$(CXX) -o prog prog.o $(LDFLAGS)	

clean:
	rm -f tests/*.result tests/*.test tests/*.result_ prog *.o


#Every .cc file in the tests directory is a test
TESTS=$(notdir $(basename $(wildcard tests/*.cc)))




#Get the intermediate file names from the list of tests.
TEST_RESULT=$(TESTS:%=tests/%.result)


# Don't delete the intermediate files, since these can take a
# long time to regenerate
.PRECIOUS: tests/%.result_ tests/%.test

#Add the rule "test" so make test works. It's not a real file, so
#mark it as phony
.PHONY: test
test:tests/results


#We don't want this file hanging around on failure since we 
#want the build depend on it. If we leave it behing then typing make
#twice in a row will suceed, since make will find the file and not try
#to rebuild it.
.DELETE_ON_ERROR: tests/results 

tests/results:$(TEST_RESULT)
	cat $(TEST_RESULT) > tests/results
	@echo -------------- Test Results ---------------
	@cat tests/results
	@echo -------------------------------------------
	@ ! grep -qv OK tests/results 


#Build a test executable from a test program. On compile error,
#create an executable which declares the error.
tests/%.test: tests/%.cc
	$(CXX) $(CXXFLAGS) $< -o $@ -I . $(LDFLAGS) ||\ { \ echo "echo 'Compile error!' ; return 126" > $@ ; \
	  chmod +x $@; \
	}

#Run the program and either use it's output (it should just say OK)
#or a failure message
tests/%.result_: tests/%.test
	$< > $@ ; \
	a=$$? ;\
	if [ $$a != 0 ]; \
	then \
	   if [ $$a -ge 128 and ] ; \
	   then \
	       echo Crash!! > $@ ; \
	   elif [ $$a -ne 126 ] ;\
	   then \
	       echo Failed > $@ ; \
	   fi;\
	else\
	    echo OK >> $@;\
	fi
	
tests/%.result: tests/%.result_
	echo $*: `tail -1 $<` > $@

#Get the C style dependencies working. Note we need to massage the test dependencies
#to make the filenames correct
.deps:
	rm -f .deps .sourcefiles
	find . -name "*.cc" | xargs -IQQQ $(CXX) $(CXXFLAGS) -MM -MG QQQ | sed -e'/test/s!\(.*\)\.o:!tests/\1.test:!'  > .deps

include .deps

The result is a basic unit testing system written in about 30 lines of GNU Make/bash. Being make based, you get all the nice properties of make: the building and testing all runs in parallel if you ask it to and if you update some file, it will only rerun the tests it needs to.The code along with some more sample tests is available here: https://github.com/edrosten/unit_tests_with_make