listmonk said the campaign finished. 24 of 40 subscribers never got it.

We gave an agent one question about listmonk, an open-source newsletter server: if it restarts while it is sending a campaign, does every subscriber still get the email? In one recorded run the answer was no, and listmonk’s status still said finished.

listmonk reports

Finished

The mail server received

16 of 40

The campaign is marked finished. The 24 subscribers who got nothing are three whole batches of eight, one after another.

The question

A restart in the middle of a send is ordinary. A deploy does it, a drained node does it, a process that runs out of memory does it. So the question is fair for any tool that sends things in batches: after the restart, does the work carry on from where it actually stopped?

There are two answers to compare. Finished is listmonk’s word for the campaign. Delivered is what the mail server holds. The check we used counts the messages the mail server received and compares them with the list of subscribers who should have got one. It never asks listmonk how things went.

The setup

The agent was given listmonk’s source code and Afterfault’s public documentation, and nothing else. It built the environment itself: listmonk v6.2.0, unmodified, its PostgreSQL database, a mail server standing in for a real provider, and a client that creates 40 subscribers and starts a campaign. It had a first target booted after 6 minutes and a first meaningful run after 7.

It then ran 170 experiments. Each one is the same campaign with a different schedule of faults: when listmonk is killed, for how long, when it is paused. The whole evaluation took 1 hour 43 minutes.

The run that was kept

The run that was kept failed the check. It has 2,849 recorded decisions over 71 seconds, and two faults:

  1. 0 sThe campaign starts. listmonk fetches subscribers in batches and sends them.
  2. 43.97 slistmonk is killed and stays down for 254 ms.
  3. +3 msA 408 ms pause follows.
  4. afterlistmonk comes back, reconnects to its database and carries on sending.
  5. endlistmonk marks the campaign finished. The mail server holds 16 messages.

The search first found the failure with three faults. Afterfault then cut it down to the two that are needed.

Why it happens

listmonk keeps a checkpoint for each campaign: the id of the last subscriber it has handed out. The query that fetches the next batch moves that checkpoint in the same statement. In queries/campaigns.sql at v6.2.0, the query next-campaign-subscribers selects the batch and also runs:

UPDATE campaigns
SET last_subscriber_id = (SELECT MAX(id) FROM subs), updated_at = NOW()

So the checkpoint means fetched, not sent. A process that is killed while batches it has fetched are still in memory starts again from the checkpoint, which is already past them. Going by the status, nothing looks wrong afterwards. The process is up, the database connection is healthy, and the campaign ends as finished.

This is a real trade-off and we do not know whether it was chosen on purpose. Moving the checkpoint when a batch is fetched means a crash can skip subscribers but cannot email them twice. Moving it after the batch is sent means the opposite. Either choice can be right. What matters is that the status reads the same in both cases, so the status alone does not tell you which one you got.

What to take from it

  1. Ask the receiver, not the sender. A status field reports what the sender believes. Count what arrived on the other side and compare it with what should have.
  2. Know which side of the work your checkpoint moves on. Before the work, a crash loses it. After the work, a crash repeats it. Decide which one you can live with and write it down.
  3. Restart at the moment work is in memory. A restart while the system is idle proves very little. The interesting restarts are the ones that land between taking work and finishing it.

Bring a system you need to trust.

Request early access Next: NATS JetStream