Byron Coke

Byron Coke

Linux infrastructure engineer. Production estates and platform migrations for banking, government and telecoms.

contracts@paxmentis.com ·Remote, UK ·Subject to availability

Thirty years in IT, more than twenty-five on Unix and Linux. Red Hat and Ubuntu at production scale, containerised platforms, high availability and disaster recovery, Ansible, and the migrations that move an estate from one to the other.

RHEL · Ubuntu · Docker · Kubernetes · Ansible · PostgreSQL · AWS · Azure · VMware · Nginx · Caddy · HAProxy · Prometheus · Grafana · ELK · TLS and PKI · SPF, DKIM and DMARC

Working with me

Two ways in. The price is on both, because you should not have to ask.

Start here

A written diagnosis · £250, fixed

You describe what it is doing and when it started. I read the logs and the config, reproduce what I can, and send you a written diagnosis within one working day: the actual cause, the actual fix, and what the fix will cost.

You can hand that to your own team and never speak to me again. That is a legitimate outcome and the price assumes it.

Paid when you have the diagnosis, not before.

The work itself

Engineering time · £600 per day

Production infrastructure: Red Hat and Ubuntu, Docker, Postgres, high availability, disaster recovery, TLS and certificate automation, and the migrations that move an estate from one place to another. Thirty years of it, across banking, government, healthcare and telecoms.

Direct engagement, invoiced through my limited company or an umbrella. Fully remote, subject to availability.

If it needs watching

Ongoing work · from £450 per month

Some systems want someone who already knows them before they break. Backups verified by restoring them rather than by a green tick, upgrades taken in a planned window, keys and credentials held somewhere other than the machine that depends on them, and a written record of how the thing is actually put together.

Scope and response times are agreed against your systems before anything starts, not picked off a tier on a web page. Tell me what falls over, how often, and what an hour of it costs you.

What I will not sell you

A retainer before I have fixed something for you. If we have not worked together, start with the diagnosis. If it turns out you do not need me again, that is a good outcome and the price assumes it.

A response window quoted off a web page. Any number published here would be a guess about systems I have not seen. The ones that mean anything get written into the contract, against your stack and your risk.

A rebuild when a fix will do. If the cheap answer is a configuration change, that is the answer you get, and it is the one I will quote for.

Things I've built

Systems I designed and built end to end. Some found a market and some did not; the engineering is the same either way. Each card plays a short piece on the architecture.

The estate

The self-hosted infrastructure everything else runs on, including this page.

Architecture and numbers

41 containers on one host. n8n in queue mode with separate workers and task runners, production and development. 20 hostnames behind one reverse proxy, 27 TLS certificates issued and renewed without intervention, which means nobody is woken at 3am for an expired certificate. 11 weeks uptime.

Docker · Caddy · Postgres · Redis · n8n · Postal · MinIO · Plausible

Outbound Command Centre

Source prospects, enrol them into sequences, dispatch across six channels, and poll every channel back for replies.

Architecture and numbers

16 workflows, 130 nodes, and a Next.js app: 198 TypeScript files, 16,009 lines, 24 Postgres tables. LinkedIn, Instagram, Email, TikTok, Google Maps and Upwork. Running live behind authentication.

Next.js · TypeScript · Drizzle · Auth.js · n8n · Postgres · Apify

Quote-to-Keys

A continuity layer for residential conveyancing, closing the gap between a quote issued and an instruction secured.

Architecture and numbers

12 workflows, 109 nodes. Intake, CSV ingest with three response paths, dormant-lead reactivation, stalled-matter nudges, missing-data alerting, scheduled weekly reporting, and an approval sequence with timeout and handler logic.

n8n · Postgres · Telegram · scheduled triggers

Grant Fit

Public IRS filings in one end, a verified funder record and a rendered document out the other.

Architecture and numbers

56 Python scripts, 9,256 lines. 7 workflows, 94 nodes. 990-PF parsing, officer extraction, lane building, prospect verification, contact discovery, then PDF render. 140 documents across 31 lanes.

Python · n8n · Postgres · WeasyPrint · Stripe · Mautic

Opportunity Layer

Screen commercial property against the public EPC register, trace each building back to the company that owns it, and produce a ranked contact list.

Architecture and numbers

32 Python files, 6,921 lines, 14 pipeline stages. Pulls the government EPC register, scores sub-standard stock, matches addresses to the registered owner through Land Registry corporate ownership data, enriches with filed Companies House accounts, and builds the contact list. Every stage is its own script: stop after any one, inspect the output, resume.

Python · pandas · EPC Open Data · HM Land Registry CCOD · Companies House API · Apify

Workspace tooling

Thirty-one small programs written to do my own work, each with one job and no shared framework between them.

Architecture and numbers

31 Python programs, 5,331 lines. SPF, DKIM and DMARC checked against live DNS with the ten-lookup limit counted; a domain screener now on its fifth version, each earlier one wrong in a way I could measure; an hourly job-board watcher that scores postings and alerts; a staleness check that flags governing documents still asserting something a later decision killed.

Python · dnspython · ffmpeg · Atom/RSS · cron

Authority Pipeline

Prospect research through to printed direct mail, unattended: qualify, scrape, write, lay out, produce the address labels.

Architecture and numbers

11 Python scripts, 2,186 lines, print-ready PDF at the far end. Qualifies a prospect list, scrapes contact details, generates a per-prospect diagnostic, renders the letter and the label sheet through headless Chrome and merges them with pypdf. The last stage in the run exists to correct what the earlier four got wrong.

Python · headless Chrome · pypdf · Anthropic API

Spotlight on Local

Local-business outreach built across n8n and Python, with mailbox warmup and a podcast generator attached.

Architecture and numbers

17 n8n workflows, 95 nodes, 5 Python programs, 1,591 lines. Prospect building with contact enrichment, nine warmup mailboxes each given its own replier workflow, a follow-up scheduler, and a FastAPI service that assembles a podcast episode. The sender went through four iterations before v4 behaved.

Python · n8n · FastAPI · reportlab · SMTP/IMAP

Selected work

Write-ups

Two incidents from my own estate, written up the way I would write them up for you: what broke, what the evidence actually said, and what I got wrong on the way.

Post-mortem

Half a million delivery attempts, nothing ever sent, and a test that passed

534,686 delivery attempts. Zero messages delivered. Every dashboard reported the system healthy.

A self-hosted Postal mail server, running in Docker on a VPS, had been accepting mail and queueing it normally for months. Messages went in. The queue looked healthy. Nothing came out the other side, and nobody noticed, because the layer that was broken was not the layer being watched.

Every attempt in the database carried the same failure:

Resolv::ResolvError

The test that made it worse

The obvious check is to resolve a domain from inside the container. I did, in Ruby, the same library Postal uses. It returned in 0.03 seconds.

That result was a false negative, and it cost real time. It had already produced a plausible and completely wrong theory, that name resolution was failing under concurrency and succeeding when tested by hand.

The manual lookup read /etc/resolv.conf. The worker read the resolver configured in Postal's own config file. Two different files. The diagnostic and the failure were never looking at the same thing.

The cause

Postal's configured resolver was 10.0.1.1. That address is the gateway of a Docker bridge network, br-e538c46466eb. It is a routable address, so nothing errors immediately, and it answers on no port at all. Nothing listens on 53.

Every MX lookup the mail server had ever made timed out against a gateway that was never a resolver. The config file was dated 8 May. The failures ran from then until September.

The fix

  • Resolver changed to 127.0.0.11, Docker's embedded DNS, which also resolves container names, with 1.1.1.1 and 8.8.8.8 behind it.
  • timeout:2 attempts:2, so a resolver failure fails fast instead of holding a worker open.
  • No root required. The config directory was owned by the service user, which meant the fix was available the whole time to anyone who had read the right file.
  • The queue was purged before the restart, not after. Half a million poisoned retries would otherwise have gone out at once against a resolver that now worked.

What I take from it

  • A dashboard reporting success is not evidence of delivery. Instrument the transport, not the application that hands work to it.
  • Check that your diagnostic reads the same configuration as the failing process. A passing test against the wrong file is worse than no test, because it sends you somewhere else.
  • A silent failure with an automatic retry can run for months. The retry is what keeps it quiet. Alert on the age of the oldest item in a queue, not on whether the queue is accepting work.

Post-mortem

Twenty-seven days with no backup, and nothing said a word

The nightly job ran every night. It reported success every night. Nothing had been uploaded since 15 August. I found out on 10 September.

The cause was not technical. A free-tier cloud account lapsed and the provider suspended it. The backup script carried on running to schedule, the uploads failed, and the failure was not loud enough to reach anyone.

Why it ran for four weeks

Every alert in the chain was watching the wrong thing. They were all asking did the job run? The job ran. It ran perfectly, twenty-seven times, to a destination that was no longer accepting anything.

Nothing was asking the only question that matters: how old is the newest backup?

The part I nearly missed

Checking the restore path afterwards, one leg was invisible on the first pass because its cron runs as root rather than as the service user. Reading one crontab is not reading the backups. Every schedule on the host is part of the estate, whoever owns it.

Honest postscript

Nothing was lost. The historical snapshots were still there because the sixty-day prune had not yet reached them. That is timing, not design. Four more weeks and the outcome is a different article.

What I take from it

  • Alert on the age of the newest artefact, not on job exit status. A job that succeeds while achieving nothing is the most common shape of silent failure I see.
  • The cause of a technical outage is frequently administrative. A lapsed account, an expired card, a rotated key nobody owned. Put a billing alarm where you think you need a monitoring probe.
  • A backup you have never restored is a hypothesis. Restore drills belong on the calendar, not in the runbook.