Two incidents from my own estate, written up the way I would write them up for you: what broke, what the evidence actually said, and what I got wrong on the way.
Post-mortem
Half a million delivery attempts, nothing ever sent, and a test that passed
534,686 delivery attempts. Zero messages
delivered. Every dashboard reported the system healthy.
A self-hosted Postal mail server, running in Docker on a VPS, had been accepting mail and
queueing it normally for months. Messages went in. The queue looked healthy. Nothing came out the
other side, and nobody noticed, because the layer that was broken was not the layer being watched.
Every attempt in the database carried the same failure:
Resolv::ResolvError
The test that made it worse
The obvious check is to resolve a domain from inside the container. I did, in Ruby, the same
library Postal uses. It returned in 0.03 seconds.
That result was a false negative, and it cost real time. It had already produced a plausible and
completely wrong theory, that name resolution was failing under concurrency and succeeding when
tested by hand.
The manual lookup read /etc/resolv.conf. The worker read the resolver configured in
Postal's own config file. Two different files. The diagnostic and the failure were never
looking at the same thing.
The cause
Postal's configured resolver was 10.0.1.1. That address is the gateway of a Docker
bridge network, br-e538c46466eb. It is a routable address, so nothing errors
immediately, and it answers on no port at all. Nothing listens on 53.
Every MX lookup the mail server had ever made timed out against a gateway that was never a
resolver. The config file was dated 8 May. The failures ran from then until September.
The fix
- Resolver changed to
127.0.0.11, Docker's embedded DNS, which also resolves
container names, with 1.1.1.1 and 8.8.8.8 behind it.
timeout:2 attempts:2, so a resolver failure fails fast instead of holding a
worker open.
- No root required. The config directory was owned by the service user, which meant the fix was
available the whole time to anyone who had read the right file.
- The queue was purged before the restart, not after. Half a million poisoned retries would
otherwise have gone out at once against a resolver that now worked.
What I take from it
- A dashboard reporting success is not evidence of delivery. Instrument the
transport, not the application that hands work to it.
- Check that your diagnostic reads the same configuration as the failing process.
A passing test against the wrong file is worse than no test, because it sends you somewhere else.
- A silent failure with an automatic retry can run for months. The retry is what
keeps it quiet. Alert on the age of the oldest item in a queue, not on whether the queue is
accepting work.
Post-mortem
Twenty-seven days with no backup, and nothing said a word
The nightly job ran every night. It reported success every night. Nothing had been uploaded
since 15 August. I found out on 10 September.
The cause was not technical. A free-tier cloud account lapsed and the provider suspended it. The
backup script carried on running to schedule, the uploads failed, and the failure was not loud
enough to reach anyone.
Why it ran for four weeks
Every alert in the chain was watching the wrong thing. They were all asking did the job
run? The job ran. It ran perfectly, twenty-seven times, to a destination that was no longer
accepting anything.
Nothing was asking the only question that matters: how old is the newest backup?
The part I nearly missed
Checking the restore path afterwards, one leg was invisible on the first pass because its cron
runs as root rather than as the service user. Reading one crontab is not reading the
backups. Every schedule on the host is part of the estate, whoever owns it.
Honest postscript
Nothing was lost. The historical snapshots were still there because the sixty-day prune had not
yet reached them. That is timing, not design. Four more weeks and the outcome is a different
article.
What I take from it
- Alert on the age of the newest artefact, not on job exit status. A job that
succeeds while achieving nothing is the most common shape of silent failure I see.
- The cause of a technical outage is frequently administrative. A lapsed
account, an expired card, a rotated key nobody owned. Put a billing alarm where you think you need
a monitoring probe.
- A backup you have never restored is a hypothesis. Restore drills belong on the
calendar, not in the runbook.