Source: Deterministic simulation testing in celld: exploring execution orders and reproducing bugs, by Yusuke Tanaka
celld runs Cloudflare Workers and Durable Objects apps on your own machines. Its team runs the production code inside a simulator that picks every event order from a seed. One seed found an alarm race no hand-written test described, and the same seed replays it on demand.
A bug in a distributed system “can depend on a particular sequence of delayed messages, failed writes, and node restarts.” Those events can arrive in a different order on the next run, so the test passes and the bug hides. Without the same order again, nobody can trace the cause or prove a fix works.
Peter Deutsch’s Eight Fallacies of Distributed Computing names the assumptions that cause this, including “The network is reliable” and “Latency is zero.”
celld separates the code that selects the next event from the code that handles it. The simulator owns selection; production code does the handling, unchanged.
Each test runs this loop up to a configured number of actions. An action is delivering a request, letting a task run, advancing the clock, or completing a storage operation.
If none remain, check completion conditions, such as every planned request having finished, and stop.
The seeded generator picks the next action, and decides whether storage succeeds or which fault fires.
celld handles the event exactly as it would in production.
After every action, the checker tests conditions the system must uphold throughout the run, using observed responses and stored data.
An alarm runs a cell’s code at a set time. celld can unload an idle cell, so it finds sleeping cells by scanning wake entries in object storage rather than opening every cell’s SQLite database. To save writes, celld reused wake entries.
celld clears the time in SQLite, saves it, and returns success. A separate cleanup task will remove the 10:00 wake entry later.
celld reuses the 10:00 wake entry to wake the cell early enough, and returns success. The old cleanup is still waiting.
It deletes the 10:00 wake entry that the new alarm still needed.
The 10:05 alarm is set in SQLite, but no wake entry can reactivate the cell. If the node restarted before 10:05, the alarm could fail to run despite the success response.
Two successes, one late cleanup, one missing wake entry.
“We had not written a test that prescribed this sequence.” The simulator found it by exploring requests and pending work.
| Bug | Lesson | Rule now |
|---|---|---|
| Reuse | A delayed cleanup for an old alarm can delete what a new alarm depends on. | Each alarm setting gets its own wake entry; old ones go only once celld confirms they are no longer needed. |
| Checks | An entry that vanishes and returns looks fine at the end. | The regression test repeats the failing order and checks the invariant during the run. |
General guidance, not celld code. The portable idea fits any system with queues, timers or retries: put every source of nondeterminism behind one seeded scheduler, and check your invariants after every step.
rng = Random(seed)
for step in range(max_actions):
actions = sim.available_actions()
if not actions:
check_completion(sim); break
sim.run(rng.choice(actions))
check_invariants(sim) # every step, not only the end
Deterministic simulation testing in celld, Yusuke Tanaka, October 2, 2026.
Sources: every fact on this page comes from the celld article above. celld is by Deno Land Inc., Apache-2.0.