Bash reads your script as it goes

I had handed a long coding task to a helper job, and sixteen minutes in it was doing perfectly fine. The job ran inside a small wrapper script that waits on the task and decides whether it has gone quiet for too long, and since I happened to be debugging exactly that part of the wrapper, I added a comment to it explaining one of the numbers. It was a harmless little comment, the kind you add so that future you knows where a number came from, and a moment later the job fell over with this:

line 172: syntax error near unexpected token `measured'

"Measured" was a word from the comment I had just written, in a file the job had loaded a quarter of an hour earlier, long before that comment existed. It found it anyway 😳

The reason is that bash doesn't load your whole script before running it. Most of us picture running a script as reading the file and then executing it, which is roughly what Python does, but bash reads a little, runs a command, and then carries on reading from the same open file at the byte position where it stopped. That position is just a number, a byte offset and not a line, so if someone changes the file underneath it the number stays exactly where it was while the text it points at quietly moves.

You can watch this happen with a three line script:

echo start
sleep 2
echo done

Start it, and while it sleeps, overwrite the file with the same script plus one comment at the top:

# tidy up the output later
echo start
sleep 2
echo done

This is what the running script printed:

start
job.sh: line 3: t: command not found
start
done

When the sleep finished, bash went back to byte 19, which used to be the start of echo done, but in the new file byte 19 lands in the middle of the comment, right at t later. So it ran a command called t, read on, found echo start and happily ran it a second time before reaching the end.

None of this is a bug, because bash is doing exactly what it promises. Read that output again though and imagine the line that ran twice was something that sends a message to a person, or deletes a folder, and it stops being a fun trick very quickly.

The fix comes from how files work on Linux rather than from trying to be more careful. A running process holds a file open by its inode, which is the file's real identity on disk, and not by its name. So if you write the new version into a temporary file and then rename it over the old one, the name now points at a brand new inode while the running bash keeps reading the old one, original bytes and all, right to the end:

tmp="$(mktemp)"
# write the new version into "$tmp" here
chmod --reference=job.sh "$tmp"
mv -f "$tmp" job.sh

The edits that bite are the ones that cut the same file down and write it again in place, like a shell > redirect, most "write this text to that file" library calls, and quite a few editors and tools. sed -i turns out to be safe, because it already writes a temporary file and renames it, which is exactly the difference that matters here.

So I changed two things. The first is a habit, which is that any shell script that might be running gets replaced by a rename and never edited in place. The second exists because habits fail on busy days: before any of my file edits go through, a small check now asks whether a running script has that file open, and refuses the edit if one does. It has more tests for false alarms than for real catches, because a guard that cries wolf on every edit gets switched off sooner or later, and then it guards nothing at all.

The part that stayed with me most was what I did in the first few minutes after the crash, which was stare at the stall detector, because that was what I had been working on and so that was where I assumed the problem was. The job hadn't stalled at all. It had been perfectly healthy right up until I changed the ground it was standing on 🙃 So now, when a long job dies, I look first at whatever I just touched, before I go looking at the thing I was debugging.