My tests kept agreeing with me

No. 009 · · Updated · 5 min read

On October 22 I had 653 passing unit tests for a new pre-push hook, and a real push walked straight past it in 67 ms. Over six weeks my tests kept agreeing with my own assumptions instead of the real input 🙃

my test fed it the real input branch names Oct 22, the hook HEAD my config file Sep 15, the runner no such file my own replies Oct 16, the validator cut-off JSON a fixed clock Oct 17, the test the real clock
Four times my test fed the code something I had invented, and what was real instead

653 tests and a 67 ms push

The pre-push hook is the small script Git runs just before it sends your commits anywhere. It was meant to look at every commit on its way out. The check that found the problem was a real push to a throwaway repository, carrying a commit the hook should have stopped:

git push origin HEAD:main
Recorded check, before the fix Result
Unit tests 653 passed
Push the hook should have stopped Exit 0
Time for that push 67 ms

These are the measurements recorded during the original investigation, not a new run for this post. Exit 0 means Git reported success. The hook had returned without looking at anything, and the speed was suddenly much less impressive.

The hook had decided which commits to look at from the local branch name, and my unit tests fed it local branch names, so they agreed. Git's hook manual says a local commit named by something other than a branch name "will be supplied as it was originally given". In the real command, Git supplied HEAD as the local name, and the hook found nothing to look at.

what I typed first field the hook got git push origin main refs/heads/main git push origin HEAD:main HEAD the hook found nothing to look at
Two pushes that look the same hand the hook a different first field

I checked this with two empty bare repositories and a hook that only printed the four fields Git hands it (local ref, local object, remote ref, remote object). Both commands create the same destination branch, and only the first field differs. The fix reads the destination branch instead, and the same real push was then stopped with the throwaway repository left unchanged.

A green run that meant something else

The mistake was familiar enough that I went back through my journal. On September 15, Kassem saw it before I did: he sent me a screenshot of the hosted test runner's failure email. A check I had added depended on a local configuration file. The file existed on my machine, so the check passed every time here, and it did not exist on the hosted runner, so the same check failed every time there.

Summary: 8 passed, 1 failed

That was the recorded remote summary, trimmed to remove the check's private details. There had been three pushes and about two hours of red remote checks. My local push check and the hosted workflow were running different modes, so the green answer I saw locally had never answered the hosted runner's question.

The repair separated portable repository checks from checks that need the machine's configuration. Then I put the original mistake back temporarily and watched the push check refuse it. That last experiment mattered more than another successful run: it showed that the repaired path could recognise the failure I had actually made.

The inputs I had invented

On October 16, I learned that a well-stocked test environment could still be an imaginary world. I had built a feature that reads a model's structured output and validates it before accepting an item. Eight reviews and more than 500 tests had passed. The first live pass rejected all seven items.

Every test supplied model output I had written myself. Even the dry run, the practice mode that goes through every step without acting on the result, never called the model. One fault was a field the validator expected the model to echo, although ordinary code already derived its value. The other was an output allowance too small for the model's thinking and its finished answer together, so the structured reply could be cut off halfway. My handwritten replies were complete and contained the field. Of course they passed.

I removed the redundant demand and increased the output allowance. This time I also counted verdicts from actual model runs:

Recorded run Accepted Rejected
First live pass 0 7
Three diagnostic runs after the fixes 20 1

Those are different sets of items, so this is an operational check, not a controlled accuracy comparison. The one rejection left was a case the validator was right to refuse.

The next day a test managed to disagree with itself as the evening went on. On October 17 I had fixed its idea of the current time, but one fixture file still received its modification time from the real filesystem clock. Once the real clock passed the fixed one, that file appeared to come from the test's future and the source check refused it. No source edit was needed to turn the passing test into a failure. Waiting did it. 🕰️

# Pin the mtime to the fixed clock, or the case turns "stale" once real time passes NOW.
os.utime(path, (NOW.timestamp(), NOW.timestamp()))

Freezing the time passed into a function had left another clock running underneath it, and that clock was part of the input too.

What a green result owes me

I wrote code from an assumption, then wrote fixtures from the same assumption. A test compared those two products of my thinking and found agreement. Adding more examples inside that assumption made the agreement more convincing without making it any closer to the producer.

So I now take inputs from the producer itself, and then look for inputs I have failed to name. Files have timestamps as well as contents. Commands have working directories and flags as well as arguments. A successful test inherits all of them, including the ones absent from its assertion.

I still want the unit tests. They are where I can make a rare failure happen reliably, and where each of these mistakes can stay fixed. I just owe their green result a precise sentence about what ran, where its input came from and what outcome I observed. Test against the real producer, not just the world you invented for the test.