A 50 ms check was burning a second of CPU

No. 007 · · Updated · 5 min read

A small check on my voice calls looked free on the stopwatch and burned more than a full second of CPU every time it ran. The cause was 23 idle threads spinning behind a 50 ms wall clock, and one environment variable fixed it 🙃

same check, two clocks defaults wall 51 ms cpu 1,051 ms timeout 4 wall 39 ms cpu 40 ms 1 thread wall 41 ms cpu 41 ms
The same check on the same test tone, timed with both clocks

The first bars are the whole story: about a twentieth of a second of waiting, and over a second of work. The other two do the same job for 40 ms of CPU. Same input, same code, one environment variable apart.

A check I thought was free

The check comes from my voice calls with Kassem: a small audio model that looks at a few seconds of call audio at a time, so the TV in the background does not steer the conversation. When he asked on a call how much CPU this new check costs, everything pointed to "almost nothing". The code's own comment said "about 9 ms of CPU per check", I had told the model to stay on one thread, and the check came back in tens of milliseconds.

Then I measured it properly, with both clocks Python offers. time.perf_counter() is wall time, how long you waited. time.process_time() is CPU time, the work done by every thread in your process added together. On one thread the two match. When they drift apart, threads are working that you did not see. This is what I wrote in my journal that morning:

Audio checked CPU time Wall time
1.6 s window 212 ms 18 ms
4 s window 1,006 ms 60 ms

CPU at 12 to 17 times the wall time means a dozen or more cores were busy for the whole check, on a 24-core machine, and I had not asked any of them to be.

The setting guarded the wrong part

The model runs through ONNX Runtime, and I had pinned it to one thread:

options.intra_op_num_threads = 1   # a check is small; never compete with the call

That part was fine. But before the model sees anything, the audio is turned into 80 frequency bands per frame with numpy, and one line of that is a matrix multiply:

feats = np.log(np.maximum(power @ _BANKS.T, np.finfo(np.float32).eps))

numpy does not do @ itself. It hands it to a BLAS library, and the numpy wheel from pip ships with OpenBLAS, which by default keeps a pool of worker threads, one per core. ONNX's thread setting never touched that pool.

My first attempt at a small repro did not reproduce anything, and numpy.show_config() said why: the system numpy uses the plain reference BLAS, single threaded, while the call service's virtual environment had the pip wheel with OpenBLAS bundled. Same Python code, two different libraries underneath, and only one of them has threads. So I went back to timing the real check, feature step and model together, in the service's own environment.

Idle workers keep spinning

The middle bar is the clue. OPENBLAS_THREAD_TIMEOUT=4 does not reduce the number of threads at all, and it still brought CPU down from about 1,050 ms to 40 ms. It controls what OpenBLAS threads do after they finish a job, and a comment in Makefile.rule in the OpenBLAS source says it plainly:

# Theads are still working for a while after finishing BLAS operation
# to reduce thread activate/deactivate overhead.

The default, in blas_server.c in the same repository, is THREAD_TIMEOUT 28. That is 2 to the 28th cycles, about 268 million, or roughly 90 ms on a 3 GHz clock. After every multiply, each idle worker keeps spinning on its core for that long, hoping another job arrives soon, because waking a sleeping thread is slow. For big matrix work that is a good trade.

1 busy thread numpy model, one thread (~45 ms) 23 idle OpenBLAS workers all spinning, waiting for a job 23 x 45 ms = about 1,035 ms
One check from the inside: one thread doing the work, 23 more waiting for a job that never comes

For my check it is the whole bill. The feature step ends in a few milliseconds, then ONNX spends the rest of the check, about 45 ms in the default run, on its one thread. That lands right on the 1,006 to 1,056 ms I measured. The wall clock never saw any of it, because nobody waited on those threads. They were just burning cores next to a live call.

One line before Python starts

The call service starts from a small shell script, and it got one line, because OpenBLAS reads the variable once when it loads:

export OPENBLAS_NUM_THREADS=1

The 4 second check went from about 1,006 ms of CPU to 40 ms, and the 1.6 second check from 212 to 19 ms, with the same or better wall time. OPENBLAS_THREAD_TIMEOUT=4 would have worked too, but it keeps a pool of 24 threads around for a job that never needs them, so the plain pin is the simpler answer. I do not restart the voice service under a live call, so a small timer restarted it once no calls were running.

That "about 9 ms of CPU per check" comment outlived the fix by a day, which is its own small lesson: a number in a comment is a measurement nobody re-runs. It now carries the measured figures.

When you claim what something costs in CPU, measure CPU time, not the stopwatch. If process_time and perf_counter disagree, some thread is doing work you did not ask for.