Queues and scheduled work

Two related things sit under this heading. A queue takes a message from the code handling a request and hands it to a consumer afterwards, so the person waiting only waits for the part they care about. A scheduled job runs on a clock with nobody waiting at all. Both swap an immediate answer for a promise that the work will happen, and both are only as good as the place failures end up.

How we use it

Objectify puts index maintenance, webhook delivery and realtime fan-out on an object-events queue with a dead letter queue behind it, so writing a record returns as soon as the row lands rather than after every downstream effect has finished. DMARC Engine evaluates alert rules on a queue and dispatches the results to email, Slack and Microsoft Teams with per-channel severity thresholds, snoozing and deduplication, and separately runs a scheduled worker that flattens SPF records and republishes them to the hosted zone so delegated includes stay current. In atHeartbeat the queue consumer lives in the same Worker as the API and sends verification and password reset email. 360nutri takes the cheapest option available to it and schedules from inside the database: pg_cron calls a recipe generation function hourly, with a control table to switch it off.

What it costs you

Delivery is at least once, so every consumer has to be safe to run twice. Sending an email is the classic handler that is not, and a duplicate welcome message is the sort of bug that reaches a customer before it reaches your logs. Either carry a deduplication key or decide out loud that the occasional double is acceptable.

Failures need a reader. A dead letter queue nothing alerts on is a bin, and a poison message can drag its whole batch through repeated retries while everything behind it waits. Depth and age are the two numbers worth putting on a dashboard on day one, because queue work is invisible to users, which means a broken consumer is usually found by the person who eventually notices the thing that never happened.

Cron does not backfill. A job that should have run at three in the morning during a broken deployment simply did not run, and nothing runs it twice to catch up. If catching up matters, it has to be part of the job's own logic. Scheduling from the database, as 360nutri does, is barer still: no retries, no dead letter queue, no alerting, and failures visible only in the database logs. It also runs whether or not anyone is looking, which in that case meant paying for model calls to generate recipes nobody had asked for.

Deciding what people cost in this shape is worth doing early, and we worked through it for a social product in which parts of a social timeline actually cost money.

When we would choose something else

For work attached to one record rather than one queue, an alarm on a Durable Object is a better tool than a cron job that scans a table looking for rows that are due. Where a system already runs a broker and needs replay, consumer groups or several services reading the same stream, we would use NATS JetStream instead, which is what the Starterflare backend does. And plenty of work does not need a queue at all: if it takes twenty milliseconds and the user wants to know it succeeded, do it in the request.