Operations

Schema drift detection

The failure this exists for is precise. The running image was built on one date, the database migrated five days later, and nobody redeployed. For three days every order died on a NOT NULL violation, /api/health returned 200 the entire time, and a buyer paid and was told "Nothing was charged."

The guard compares the columns the code writes against what information_schema actually reports:

Surface	Behaviour on drift
Boot	Banner in the deploy log and the operator page
/api/health	503, status: degraded, checks.schema: "drift"
Order endpoints	503 schema_drift before settlement, so no charge
Worker	Stops claiming, pages once, resumes by itself

Checked per request (cached 60s), not only at boot. The outage above happened under a process that had started cleanly, so a boot-only check would have passed and then been wrong for three days. The cache TTL also means the service recovers on its own once the missing deploy lands.

An unreachable database is deliberately not treated as drift. That is a different failure with a different fix. Refusing every payment because one catalog read timed out would be its own outage.

Deploy traps

Auto-deploy is on: pushing to main redeploys web, worker and cron.

pnpm verify does not catch build failures

verify runs typecheck, lint and the full suite. All three passed on a commit whose production build failed outright.

instrumentation.ts is compiled for the Edge runtime as well as Node. A static import there that reaches node:crypto, node:dns or Prisma fails webpack with UnhandledSchemeError. Nothing but a real build sees it.

pnpm build is not optional and verify does not replace it. Ninety seconds locally beats a five-minute round trip to a builder.

railway up reports success without deploying, in two distinct ways

1. --detach returned exit 0 twice without deploying anything. No error, no warning, the old image still serving. Attached railway up worked first try. Exit 0 from --detach is not evidence of a deploy.

2. SKIPPED. If a build fails for an infrastructure reason and you re-run with the same source, Railway compares against that failed deployment, decides nothing changed in the watch paths, and records the new one as SKIPPED. It does this cheerfully, at exit 0, still running the last image that succeeded.

   SKIPPED is often correct: web declares watch patterns, so a documentation-only push rebuilds nothing. It is a problem only when you did change a watched file.

The check that settles it either way is .build.commitSha from /api/health. If it matches the commit you pushed, that commit is serving, whatever the deployment record says.

Migrate before pushing

The database and the code must move together, and a push only moves the code:

pnpm verify
pnpm build          # NOT optional
pnpm db:deploy      # migrate FIRST: new code expects the new schema
git push origin main

Migrating after the push inverts the outage rather than fixing it.

lint:dead

Fails the build when an exported function or class is reachable only from tests.

It exists because a subsystem gets built and tested, and then nobody wires an entry point. Nothing in the toolchain notices, because a green suite looks identical either way. That produced 92 unreachable exports, including the old Lit vault backend (then called non-custodial), the cash advance matcher, and the adapter-health alerter the runbook claimed would page the operator.

Reachable means referenced in src/ (including its own file: a helper used by its own module is not dead), or in scripts/, or a framework entry point.

Two escape hatches, both requiring a stated reason:

• @wiring-exempt: <reason> on the export. "We'll need it later" is a fine reason when true. The point is that it is written down and reviewable.
• A baseline file holding the existing backlog, so the check could land without wiring 92 things first. It fails on anything new and only shrinks.

A subtlety that cost a false negative

The first version counted a mention in a comment as a reference. So an export merely discussed in prose looked reachable, and the check could have been silenced by writing a sentence about the thing you failed to wire. It strips comments and string literals before counting. That correction moved the true count from 78 to 92.

Test-suite hygiene

The suite made live outbound calls with production credentials for most of the project's life. tests/setup.ts did delete process.env.RESEND_API_KEY to stop tests emailing. But the dotenv loader fills any key that is undefined, so the delete was exactly what it repopulated. Every refund test hit the real Resend API. The calls were rejected at their end, which is why it read as stderr noise rather than as traffic.

Assigning '' survives the loader and reads as unset to the capability probes.

Also: never run two suites against the same test database concurrently. They truncate each other's tables, and the resulting failures look like real regressions.