How an empty-string compare almost shipped our Mac app with unreadable crash reports

If a customer sent your team a crash report today from a build you shipped in spring, could someone turn it into function names and line numbers within the hour? On October 19 we had to answer that for keychord, a Mac keyboard tool we are building, because we had just made its daemon smaller by throwing away the very data that answer depends on.

That morning the daemon went from 4,057,264 bytes to 2,584,576, about 36% smaller. Three release settings did it, namely whole-program link-time optimisation, a single code generation unit, and stripping the symbol table out of the shipped binary.

That last setting has a cost most teams only discover during an incident. A stripped binary still crashes like any other program, but the crash report it leaves behind is a list of raw memory addresses. Nobody on call can read "0x1000a3f20". To turn those addresses back into function names and line numbers you need the symbols you just threw away, and you need the exact copy that matches the build the customer is running.

So the same morning we changed how Mac builds are made. We added a second release profile that inherits every size setting and also records line tables. The Rust compiler runs Apple's dsymutil before it strips, so the shipped binary is the same stripped size, and a separate .dSYM symbol bundle lands next to it. The two share a Mach-O UUID, a build fingerprint that macOS also prints in every crash report under "Binary Images". The bundle never goes inside the app. A small script, save-symbols.sh, copies it into a folder named after the version and the git commit.

Before the script copies each bundle, it compares the UUID of the binary with the UUID of that bundle. If they differ, the release step fails. A symbol file from a neighbouring build would resolve addresses to the wrong functions, and a wrong stack trace is worse than none because people act on it.

Four hours later we found a hole in that check. The script read each UUID with dwarfdump and compared the two strings. If dwarfdump printed nothing for both files, the two empty strings would compare as equal. The check would have passed on exactly the build it was written to catch. We added one check that refuses a binary with an empty UUID before the comparison runs. That commit is d591be1 in our repo.

The third problem showed up that afternoon. We had moved every Rust project on the build machine to one shared cargo target directory. The symbol folders lived inside the target directory, and target directories get cleaned. One routine cleanup would have deleted the local copy of every shipped dSYM, which was the only one kept for local Mac builds. We moved them to a fixed folder outside it, overridable with one environment variable, and wrote down why in the script header and in docs/symbolicate.md.

That document is short on purpose. It says where the symbols for a given version live, how to confirm a dSYM matches the crashed binary with one dwarfdump command, and how to resolve an address with atos. The person reading it is whoever is on call when the first real crash report arrives, and they will not have the context of the day it was written.

Nothing here is exotic. Every platform has its version of this, with dSYMs on Apple, PDB files on Windows, separate debug info on Linux, source maps for minified JavaScript. In every case the same three questions decide whether a production crash costs ten minutes or two days. Is the symbol file kept for every build that ships? Is it provably the one that matches? Does it live somewhere a cleanup job cannot reach?

We answered them before keychord has a single outside user, because size work and crash readability were the same change and we wanted them reviewed together.

A few questions for your own team. When you last made a release smaller or faster, did anyone check what it did to your crash reports? If a customer sent you a stack trace from a build six months old, could you resolve it today? And who would notice if a check like ours compared two empty strings and passed?

If you want help putting this kind of release hygiene in place, connect with us: deemwar.com/consulting