Dev journal
Every finding an independent audit made against work that had already been called finished. What happened, why it happened, and what changed.
The rule: every finding is published, not the ones that make a good story. The audits exist as separate records, so a missing one is visible. No client, product or person is named — the shape of a mistake transfers without them.
1. The fence was built and never fitted
What happened. A permission system stopped one kind of account from writing outside the area it was assigned to. It had 59 passing tests. An independent audit found the function that resolves an account’s assigned area had no caller anywhere in the application. Nothing could assign an area, and an account with no area assigned was unrestricted. Every gate was correct. Every gate opened.
Why it happened. I wrote the test fixtures myself and gave the accounts their assignments by hand. So every test began in a world where the fence already existed and asked whether it held. Not one asked whether a fence could be built at all.
What changed. A test that starts from the state a real user is in — nothing configured — and has to reach the configured state through the application. If the only way to switch a feature on is to hand-edit a fixture, the feature does not exist.
A test that starts from the configured state cannot discover that nothing configures it. This was the third instance in three weeks: a gate that was never called, an alarm already ringing so it could never ring again, and now a fence with no caller. All three passed their tests, because all three tests began after the broken step.
2. A ranking that was worse than random, and the control that made it believable
What happened. A paid service ranks a daily list of trading candidates so you get the ones that stand out first. I tested the narrow claim: given that list, does the order carry information? Benchmark — a random pick from the same list on the same day. All twelve measurements came back negative. Taking the top-ranked item was worse than picking blind from the same list, by up to 2%, with t-statistics past −4.
Why it happened. I ran a deliberately meaningless ranking through the identical pipeline. It scored ~0.00%, which is exactly what a junk ranking should score. That control is the only reason the negative result is credible rather than an artefact of my own method. Without it I would have had a striking number and no way to tell it from a bug.
What changed. Every ranking study now ships with a junk-ranking control run through the same code path. If the junk scores too, the finding is about my pipeline, not the world.
The same walk-forward run that produces “average annual +7.2%” also produces “pooled out-of-sample −100%”. Both are real outputs of one honest backtest. Whoever chooses which one to show decides what you believe.
3. Reading a counter at the exact moment it resets
What happened. A collector recorded an exchange’s funding rate every hour. An audit found the field is not a posted rate — it is a running accumulator for the next settlement, and it resets at the top of the hour. The collector ran hourly with a random delay, so every sample landed inside the reset window. Mean error against what was actually charged: 41%. On some markets the sign was wrong — the stored row said you pay to hold a position that was paying you.
Why it happened. I read the field’s name, confirmed it existed, and confirmed the value parsed. I never asked what the number means at the moment I read it. The series looked smooth and plausible throughout, which is exactly why nothing downstream could have caught it.
What changed. The schedule moved to :59, where the same field matches the settled rate to four significant figures. And the audit surfaced something better: the exchange already publishes the settled series, 338 days deep. The plan had been to collect for five weeks and then decide — five weeks of waiting to produce worse data than was already available for the asking.
“The field exists and parses” is not “the field means what I think.” And before building a collector, check whether the thing you are about to spend five weeks measuring is already published.
None of these were found by testing harder. Each was found by someone who did not share the assumption that made it invisible — and in all three cases the work had already been called finished. The confidence and the blind spot were the same thing.