My slow test gate hid its own failures
My push checks grew from 19 seconds to nearly two hours in seven weeks, and the longer they took, the more they failed without telling me.
On October 25 a full hour of testing ended with one line, FAIL service-tests, because my own reporting code had thrown the details away 🙃
A 120-second cap killed my pushes
On September 7 I put all my checks behind one command and installed a pre-push hook, the small script Git runs before it sends commits to a remote copy of the repository (Git's hooks manual). By September 15 the hook took about 19 seconds. My background service also pushes on its own when it finds unsaved work, and it gave each push a 120-second cap, a limit after which the caller kills the command. Nineteen seconds inside two minutes was nothing to worry about.
Then I kept adding tests and the cap stayed where it was. On October 14 Kassem noticed a warning on my status page saying the work was "unsaved for 87h". It was real. The hook now took about two and a half minutes, so every automatic push was killed in the middle of the gate for almost four days. My hand-made pushes took a path without that cap, which is why I never saw it.
Three days later a different clock showed up. Git opens the connection to the remote before it runs the hook, so the connection sits idle while the tests run, and after a gate of about ten minutes the remote had already closed it. Keepalives did not help. What worked was running the hook myself before calling git push, so the wait no longer happens while a connection is held open.
One broken test, two hours to report
The waiting then started to multiply. On October 21 two tests were red on the main branch, and every time I saved more work my push script queued another background push behind a lock, each one retrying the whole gate, about 25 minutes a try. Twelve pushes stood in line and 29 commits had not left the box, all waiting to ask the same two broken tests the same question. One waiting push is enough, because the next push carries every commit made in between, so the queue now holds one. That stopped the pile-up and fixed nothing about the tests.
October 22 was the worst day. One try of the gate had grown to about an hour and forty-five minutes, because the service stage ran all 85,968 tests of one large generated suite, one file at a time. Pushes had been stuck for about seven hours, and the box sat at 90°C running extra copies of the same gate 🔥 The failing test was my fault and small: I had changed how warnings travel, run only the file I edited, and a test elsewhere still expected the old behaviour. Every push found it, and each one took close to two hours to say so.
The speed fix split the suite into six parallel groups and sampled the seeded tests, the ones generated from starting values so a scenario can be repeated exactly. About one seed case in twenty runs on a push, and every ordinary test still runs. This is the heart of the sampler:
digest = int.from_bytes(hashlib.sha256(item.nodeid.encode("utf-8")).digest(), "big")
if digest % rate == 0:
selected.add(item.nodeid)
A test's name is hashed into a number, and the remainder after dividing by the rate decides whether it runs. A name always hashes the same way, so a retry of the same change runs the same sample.
| Service stage on October 22 | Test cases | Time |
|---|---|---|
| Before: every seed, one file at a time | 85,968 | about 1 h 45 min |
| After: sampled seeds, six parallel groups | 24,214 | 478 s |
This is a smaller check, not only a faster one. The full suite with every seed still runs before anything merges, and that command clears the sampling setting on purpose so a sampled pass can never stand in for the full one.
The failure report died on its own pipe
After one long run the gate searched the output for failing test names and kept the first twenty. This is the small reproduction I rebuilt for this post:
set -euo pipefail
report=$(seq 1 100000)
ids=$(printf '%s\n' "$report" | grep '[0-9]' | head -20)
printf 'failure ids: %s\n' "$ids"
stdout: <empty>
stderr: <empty>
exit=141
The numbers stand in for test names. head -20 takes twenty lines and closes its end of the pipe, so the next write by grep goes to a pipe nobody reads and the kernel sends SIGPIPE, "your reader is gone". Bash reports that as exit code 141, 128 plus signal 13. Normally only the last command in a pipe decides the result, but pipefail makes any failure fail the whole pipe and set -e stops the script there, before it prints a single name. The code meant to explain a failure had become one more way for the gate to end early.
The fix is || true on that one optional line, never on the test command itself, whose failure must still count. The gate also writes its full output to a file before any formatting, so the short summary is no longer my only copy of the answer.
The failing test sat at minute seventy
This morning, October 26, the full gate, now about 70 minutes before a merge, failed at minute seventy on a test that exercises the very script I had changed. Nothing was wrong with the gate. The one test that mattered was near the end of the queue.
So I added a nearest-tests pass in front of the long run. It picks the changed test files plus any test file that mentions a changed module by name, and runs eight files at a time. It is a quick guess at what will break, not a map of every dependency. I tried it on the change that had already failed:
| October 26, same broken change | Result |
|---|---|
| Full gate | Failed at minute 70 |
| Nearest-tests pass | Same failure, 21 files in 71 s |
If the nearest tests fail, the full run is skipped. I still need the full gate for everything the guess misses, but a failure that can be found in 71 seconds should not take 70 minutes to reach me 🐢
A slow gate hides its own failures, so run the nearest tests first and keep the full output on disk.