Our automation said "done". The work had failed, on the wrong machine.

Most small companies now run on automated jobs nobody watches closely. A nightly check, a weekly report, a script that confirms the laptops and servers are set up right. When one of them shows a green light, people move on. On October 14 we found one of ours had been showing green while the work behind it had failed.

The job is called prepare-worker. It runs inside the workflow tool we build and use to run our own agents. It visits each of the machines that do our heavier work and reports which tools and logins are in place, so we know which machine can take which task.

Its settings file named the machines to visit with a key written runs_on. Our tool expects runs-on, with a hyphen. One character. Our tool read the file, did not recognise the setting, and ignored it without saying anything. With no machines named, every step ran on our main server instead. The report it produced was accurate, for the one machine we were not asking about.

That would still have been caught if the run had looked broken. It did not. One of its steps failed outright, and the run still said "done". At the time, our tool treated a failed command as information for later steps to read, and counted it as a finished step. We even had a test confirming that behaviour.

What a false green costs

A red light gets attention within minutes. A green light gets trust. Someone reads "done", assumes the machines are ready, and assigns work to them. If the work then breaks, the team starts looking in the wrong place, because the one report that should explain it says everything is fine. That is how a small mistake turns into lost days: people debugging the task, the person, or the vendor, when the real problem was a check that never ran where it claimed.

For a company of ten or twenty people, those days come straight out of the week. There is no separate ops team to notice that the numbers look too clean.

What we changed

We changed both behaviours that same morning.

A failed step now fails the whole run. The output is kept, so someone can read why, and any step that depended on it stops. If a step is designed to report a problem for a later step to act on, the file has to say so in writing.

A setting our tool does not recognise is now an error, with the name of the setting and the line it is on. The check runs when the file is loaded and in the validate command a person runs before switching a workflow on. The prepare-worker file itself now reads:

runs-on: self-hosted

with a comment above it explaining what the old spelling did.

Two minutes later we fixed the first thing the stricter rule would have broken. Our production watchers use a specific failure code to mean "something is unhealthy, start the triage step". Under the new rule, that signal would have stopped the run at exactly the moment we needed triage. The tool now spots when a later step is waiting to read an earlier step's result, and lets that signal through. We added a test that loads a watcher shaped like our real one and checks that its alert still reaches triage.

In all, the change added twelve tests. The first one opens with a comment that sums up the whole story: "prepare-worker exited 1 and the run said done."

The takeaway

Most automation tools are built to be forgiving. They ignore settings they do not understand and keep going when a step fails, because that keeps old setups working. Forgiving tools are pleasant until the day they hide a real problem behind a green light.

We now prefer a tool that complains loudly about a typo over one that guesses. It costs a few minutes when someone writes a setting wrong. It saves the days spent trusting a report that was never true.

If you run a small team, ask whoever set up your automation two questions. If someone misspells a setting, does anything complain? Can a job fail and still show green? If the answer to either is "not sure", give it an afternoon this month.

If you want help making your own automation tell the truth, connect with us: deemwar.com/consulting