An analytics write
took down our lead form.

01 · Symptom

The symptom.

Run a scan, see the summary, enter an email to unlock the full report. The endpoint that handles that submission returned HTTP 500. The front end did the right thing with a 500 — it showed a failure message and invited the visitor to try again — and trying again produced the same 500, because the failure was completely deterministic.

Everything upstream worked. Scans ran. Reports rendered. The homepage was fine. It was one endpoint, at the exact point where an anonymous visitor turns into someone we can talk to.

02 · Diagnosis

The write worked. The request failed anyway.

The handler does four things in order: validate the email, apply a per-IP limit, save the lead, then record an event row so that a scan, a lead, and a later email send can be traced end to end.

Step three succeeded. The email was in the database. Step four threw, because the table it writes to had never been created in the production database — the schema change shipped with the code, but the migration that applies it to production is a separate, manual step, and it had not been run. The exception propagated, the runtime turned it into a 500, and the response the visitor received said the whole thing had failed.

So the state of the world after every one of those requests was: we had what we needed, and we told the person we did not. Both halves are damage. The visitor never saw the full report they had just paid for with an email address, never saw what we do, and never got a route to contact us. And the record we did keep was one we did not know we had.

The most dangerous failure is the one where the important work succeeded and the response says otherwise.

03 · Root cause

Two bugs, not one.

The easy read is “someone forgot to run a migration.” True, and not the interesting part — that will happen again, to us and to you, because manual steps get skipped. The real defect is that a missing migration was able to take down lead capture at all.

Look at what those two writes are for. One of them stores a person who wants something from us: business-critical, must not be lost, cannot be reconstructed. The other stores a row for tracing and counting: useful, nice to have, and reconstructible from logs. We had written them as if they were the same kind of thing, in sequence, with the same error semantics, both able to decide the status code.

That is a severity mistake, not a database mistake. Once you see it in these terms it is obvious which of the two is allowed to fail: the one whose loss costs nothing.

04 · The fix

The fix is four lines.

The lead write stays exactly where it is and stays able to fail loudly — if we cannot store the email, the honest answer is an error. The event write is wrapped so that it can never affect the outcome: try, catch, log the failure with enough context to find it later, continue. The response is built from the state of the lead, not the state of the analytics row.

Three principles came out of it that we now apply everywhere:

• Commit what matters first. Order the work so that the request has already achieved its purpose before any optional step runs.
• Optional work cannot change the status code. If a step is allowed to fail, it must be wrapped where it is called. “It will never throw” is not a design.
• Log the swallowed failure. A silent catch converts a loud bug into a slow one. The log line is what makes the trade-off honest — lossy, not blind.

We also left the reasoning in a comment above the code rather than in a ticket, so the next person to touch that function is told why the catch is there and does not tidy it away.

05 · Generalise

The same shape, in a store.

We publish this because we have found the identical shape inside client stores more than once, and it is always worth more than the hour it takes to look for it.

On the thank-you page. A tracking or marketing script that throws during checkout completion, taking a bit of the page with it. The order is placed — the money is safe — but the customer sees something broken at the exact moment you most want them calm, and your reporting is wrong too. Anything running there should be defensive by default. If you want to know what is actually firing on your own checkout, our tracking-stack check at pixel.googogogo.com is free and reads only public pages.

In a webhook consumer. An endpoint that processes an order and then writes an audit or analytics row. If the audit write throws, the endpoint returns an error, Shopify treats the delivery as failed and retries it — and if it keeps failing, delivery stops. A nice-to-have row silently disables your order pipeline. Acknowledge first, do the optional work after.

In an app or theme integration. Any place where a display concern and a commercial concern share a code path and an error handler. If a review widget can prevent an add-to-cart, that is the same bug with a different costume.

The general test is one sentence long: for every step in this request, what happens to the customer if only that step fails? If the answer for a step that does not matter is “the request fails,” you have found it.

06 · Process

Why the tests passed.

The uncomfortable part. This code had a real test suite, and the suite covered this endpoint, and every test passed — before the incident and after it. The tests run against an in-memory store that implements every method the code calls. A test double is by definition a thing that exists. It cannot tell you that the production table does not.

So we changed two things about how we work rather than only fixing the code. First, a schema change is not done when the file is committed; it is done when it has been applied to the environment the code will run in, and that step now sits in the deploy checklist next to the deploy itself. Second, the post-deploy check exercises the funnel rather than the health endpoint. A health check tells you the process is alive. It does not tell you the one path that earns you a customer is returning a 500.

That second one generalises to every store we build: the smoke test after a launch has to include a real add-to-cart and a real checkout start, not a homepage that loads.

Quick check

Go and look for this today.

• Optional writes — analytics, audit, notification — are wrapped and cannot fail the request.
• The business-critical write happens first, and the response reflects it.
• Every swallowed error is logged with enough context to find it.
• Schema migrations are on the deploy checklist, not in someone’s head.
• The post-deploy smoke test submits a form and starts a checkout, not just loads a page.

Build logs should include the bad days.

A studio journal that only contains launches is marketing. The reason we would rather write this one up is simple: you are going to hand someone access to the thing that takes your money, and the only useful signal about how they will behave when something breaks is how they talk about the last time it did.

The tools these entries come from are free and need no signup — the readiness scan at scan.googogogo.com, and the others listed on our tools page. That unlock path has been fixed. If anything there breaks again, tell us and we will write that one up too.

Questions

Questions people ask.