Code coverage is one of those wacky metrics that straddles the line between useful and vanity. On one hand, it gives you an idea of how safely you can make changes, but on the other it can be a complete fake-out depending on how the tests are constructed. And it can slow your build down.

A lot.

I suspect a lot of our pain is self-induced, but our Laravel application’s ‘build and package’ job jumps from under 2 minutes to around 15 once we turn on code coverage. Ouch.

So I came up with a compromise position in the build in that the tests always run, but the coverage only gets calculated every 15th build (around once a day). Here is what the relevant task for that job now looks like.

<pre lang="sh"># hack around jenkins doing -xe
#set +e

mkdir -p jenkins/phpunit
mkdir -p jenkins/phpunit/clover

# run coverage only every 15 builds
if [ $(($BUILD_ID%15)) -eq 0 ]; then
  phpunit --log-junit jenkins/phpunit/junit.xml --coverage-clover jenkins/phpunit/clover.xml --coverage-html jenkins/phpunit/clover
else
  phpunit --log-junit jenkins/phpunit/junit.xml
fi

# hack around presently busted test
#exit 0

Some things of note;

  • The commented out bits at the beginning and end allow me to force a clean build if I really, really want one
  • My servers are all Ubuntu so use ‘dash’ as its shell which forces slightly different syntax which my fingers never get right the first time
  • I don’t delete the coverage log so the later ‘publish’ action doesn’t fall down. It just republishes that again
  • As we hire more people and the frequency of things landing in the repo increases, I’ll likely increase the spread from 15 to something higher
  • At some point we’ll spend the time to look at why the tests are so slow, but not now.