What a year of this has taught me about failure
Reading back over these posts, the pattern is hard to miss.
The widget id paired with the wrong app: nothing reported it. The private tags: every layer reported success and the page showed nothing. The embedded script: a non-OK response treated as “nothing to show”. The bare 403 with an empty message. The overheads billed three times, each request reporting the same balance afterwards. The profit figure whose label and arithmetic agreed with each other and were both wrong.
Almost every expensive bug in either project reported success.
The genuinely broken things — a crash, a 500, a red test — cost hours. The things that cost days or weeks all looked fine. They looked fine to the logs, to the tests, and to the person who wrote them.
I do not think the answer is more tests, exactly. Two of the worst were invisible to any test that could have been written in the environment the tests run in: the money column that overflowed only on Postgres, and the font type that was wrong only in the container. Both were eventually caught by asserting something about the declaration rather than the behaviour — the column type, the MIME registration.
So the working rule I have ended up with is this: when something does not work and nothing is complaining, stop debugging the code and go and establish ground truth somewhere outside it. Request the URL yourself. Read the schema. Check what the container actually serves, not what the test client does.
The corollary is less comfortable. A green test suite means the things you thought of are still true. It says nothing whatever about the things you did not.