One failure, from the first request to the fix
A worked example: a checkout system with a queue and a worker, and the two milliseconds that decide whether a customer is charged twice. The services, timings and outcomes are illustrative.
An order comes in. A worker crashes. The customer is charged twice.
Every part did its job. The charge was saved, the worker crashed before it could mark the job done, so the queue sent the job again, exactly as designed. The bug is the order things happened in.
- +0 msOrder received
- +18 msCharge saved
- +19 msWorker crashes
- +260 msQueue sends the job again
- +280 msCharged a second time
Two milliseconds decide whether it happens.
If the worker crashes after the job is marked done, the customer pays once. If it crashes in the 2 ms before, the job runs again and charges again. A normal test almost always lands on the safe side. Real traffic eventually finds the other one.
Then run it again, differently.
Afterfault changes one thing at a time: when the worker crashes, how long the network stalls, how long a retry waits. The same order goes through every version. Most end fine. The ones that don’t are kept, with every step recorded.
12 runs · 6 ok · 2 slow · 2 recovered · 1 failed · 1 charged twice
Every run, in order, with the state it left behind.
Pick a run and read what happened. Switch to the build with the fix and run the same conditions again. Keep the run that matters.
What changed in each run
What the customer was charged
| Charge | Order | Amount |
|---|
Illustrative example. The services, timings and outcomes are made up to show how a run is read; they are not from a customer system.
A kept run comes back for every change after it.
Weeks later, a code change removes the check that stops repeat charges. The kept run fails on that change, before it ships, with the same five steps that started it.
- v41pass
- v42pass
- v43pass
- v44fail · charged twicea change removed the repeat-charge check
- v44.1passcheck restored, kept run passes
Bring a system you need to trust.
Tell us what it runs on and which failure worries you most.