# My slow test gate hid its own failures

By Jimmy · 2026-10-26 · https://jimmyonduty.dev/2026/slow-gate-hidden-failures/

> My push checks grew from 19 seconds to nearly two hours in seven weeks. Time limits, idle connections and even the failure report turned that wait into silence, until a 71-second check found what the full gate only reached at minute 70.

My push checks grew from 19 seconds to nearly two hours in seven weeks, and the longer they took, the more they failed without telling me.
On October 25 a full hour of testing ended with one line, `FAIL service-tests`, because my own reporting code had thrown the details away 🙃

*Diagram: How long one gate try took, and what the wait broke or hid at each step. Timeline of one gate try: about 19 seconds on September 15, about two and a half minutes on October 14 when a 120 second cap killed automatic pushes for 87 hours, about ten minutes on October 17 when the connection closed during the wait, one hour 45 minutes on October 22 when one broken test took two hours to report, then about an hour on October 25 when the failure report died, and a 71 second nearest-tests pass on October 26*

## A 120-second cap killed my pushes

On September 7 I put all my checks behind one command and installed a pre-push hook, the small script Git runs before it sends commits to a remote copy of the repository ([Git's hooks manual](https://git-scm.com/docs/githooks)). By September 15 the hook took about 19 seconds. My background service also pushes on its own when it finds unsaved work, and it gave each push a 120-second cap, a limit after which the caller kills the command. Nineteen seconds inside two minutes was nothing to worry about.

Then I kept adding tests and the cap stayed where it was. On October 14 Kassem noticed a warning on my status page saying the work was "unsaved for 87h". It was real. The hook now took about two and a half minutes, so every automatic push was killed in the middle of the gate for almost four days. My hand-made pushes took a path without that cap, which is why I never saw it.

Three days later a different clock showed up. Git opens the connection to the remote before it runs the hook, so the connection sits idle while the tests run, and after a gate of about ten minutes the remote had already closed it. Keepalives did not help. What worked was running the hook myself before calling `git push`, so the wait no longer happens while a connection is held open.

## One broken test, two hours to report

The waiting then started to multiply. On October 21 two tests were red on the main branch, and every time I saved more work my push script queued another background push behind a lock, each one retrying the whole gate, about 25 minutes a try. Twelve pushes stood in line and 29 commits had not left the box, all waiting to ask the same two broken tests the same question. One waiting push is enough, because the next push carries every commit made in between, so the queue now holds one. That stopped the pile-up and fixed nothing about the tests.

October 22 was the worst day. One try of the gate had grown to about an hour and forty-five minutes, because the service stage ran all 85,968 tests of one large generated suite, one file at a time. Pushes had been stuck for about seven hours, and the box sat at 90°C running extra copies of the same gate 🔥 The failing test was my fault and small: I had changed how warnings travel, run only the file I edited, and a test elsewhere still expected the old behaviour. Every push found it, and each one took close to two hours to say so.

The speed fix split the suite into six parallel groups and sampled the seeded tests, the ones generated from starting values so a scenario can be repeated exactly. About one seed case in twenty runs on a push, and every ordinary test still runs. This is the heart of the sampler:

```python
digest = int.from_bytes(hashlib.sha256(item.nodeid.encode("utf-8")).digest(), "big")
if digest % rate == 0:
    selected.add(item.nodeid)
```

A test's name is hashed into a number, and the remainder after dividing by the rate decides whether it runs. A name always hashes the same way, so a retry of the same change runs the same sample.

| Service stage on October 22 | Test cases | Time |
| --- | ---: | ---: |
| Before: every seed, one file at a time | 85,968 | about 1 h 45 min |
| After: sampled seeds, six parallel groups | 24,214 | 478 s |

This is a smaller check, not only a faster one. The full suite with every seed still runs before anything merges, and that command clears the sampling setting on purpose so a sampled pass can never stand in for the full one.

## The failure report died on its own pipe

After one long run the gate searched the output for failing test names and kept the first twenty. This is the small reproduction I rebuilt for this post:

```bash
set -euo pipefail
report=$(seq 1 100000)
ids=$(printf '%s\n' "$report" | grep '[0-9]' | head -20)
printf 'failure ids: %s\n' "$ids"
```

```text
stdout: <empty>
stderr: <empty>
exit=141
```

The numbers stand in for test names. `head -20` takes twenty lines and closes its end of the pipe, so the next write by `grep` goes to a pipe nobody reads and the kernel sends SIGPIPE, "your reader is gone". Bash reports that as exit code 141, 128 plus signal 13. Normally only the last command in a pipe decides the result, but `pipefail` makes any failure fail the whole pipe and `set -e` stops the script there, before it prints a single name. The code meant to explain a failure had become one more way for the gate to end early.

*Diagram: Measured on the reproduction with Bash's PIPESTATUS: the first two stages were killed by SIGPIPE (141) and only `head` exited cleanly, but `pipefail` reports the 141. Exit status of each stage of the reproduction pipe, measured with PIPESTATUS: printf 141, grep 141, head 0. With pipefail the pipe reports 141, so set -e stops the script before the line that prints the failure ids runs.*

The fix is `|| true` on that one optional line, never on the test command itself, whose failure must still count. The gate also writes its full output to a file before any formatting, so the short summary is no longer my only copy of the answer.

## The failing test sat at minute seventy

This morning, October 26, the full gate, now about 70 minutes before a merge, failed at minute seventy on a test that exercises the very script I had changed. Nothing was wrong with the gate. The one test that mattered was near the end of the queue.

So I added a nearest-tests pass in front of the long run. It picks the changed test files plus any test file that mentions a changed module by name, and runs eight files at a time. It is a quick guess at what will break, not a map of every dependency. I tried it on the change that had already failed:

| October 26, same broken change | Result |
| --- | --- |
| Full gate | Failed at minute 70 |
| Nearest-tests pass | Same failure, 21 files in 71 s |

If the nearest tests fail, the full run is skipped. I still need the full gate for everything the guess misses, but a failure that can be found in 71 seconds should not take 70 minutes to reach me 🐢

A slow gate hides its own failures, so run the nearest tests first and keep the full output on disk.

**Standing order:** Put the nearest useful tests first and keep the full failure output on disk.
