Why we rebuilt our model around a mistake we didn't have to tell you about.
In the last post I wrote that this platform requires "a different kind of intellectual honesty - the willingness to say, in public, that a model didn't pass its own evaluation criteria." I didn't expect to test that promise quite so soon.
Since that post, the most important work we've done wasn't adding a feature or launching a new market. It was discovering that our single most important model input was, in a specific technical sense, lying to us - and then doing something about it.
I want to walk through what happened, because it's the clearest example I have of what we actually mean when we say transparency.
The feature that was too good
Every model has the inputs it leans on hardest. For ours, the standout was a bullpen-quality metric: a measure of how effective a team's relief corps had been, built from pitch-level data. Across every target the model predicted, it kept surfacing as the dominant signal. The story practically wrote itself: bullpens win baseball games, and our model had figured that out before the market did.
That story was wrong. Not because bullpens don't matter - they absolutely do - but because the feature wasn't measuring what I thought it was measuring.
The metric was subtly weighted by how much of a game had already been played. A reliever's contribution looks different in the seventh inning than it does before first pitch, and the way the feature was constructed let a sliver of within-game information bleed into a number that was supposed to be available before the game started. In machine learning, this is called leakage: the model gets a peek at information it wouldn't actually have at the moment it makes a prediction.
Leakage is the most seductive kind of error, because it doesn't make your model look broken. It makes it look brilliant. The model wasn't predicting games better. It was, in a small way, being handed part of the answer.
Why you have to suspect your best feature
Here's the uncomfortable part. The leak hid inside our most important feature, and that is exactly where leaks like to hide. A feature that dominates every prediction is either a genuine, powerful signal or a leak, and from the outside the two are identical. Both push your accuracy up. Both make you feel smart.
The only way to tell them apart is to interrogate your best feature the hardest; to be most suspicious of the result you most want to be true. That's an unnatural instinct. The natural instinct is to admire your best number and move on. We caught this because we went looking for trouble in the place we least wanted to find it.
When we de-leaked the feature - rebuilt it so it could only ever use information available before first pitch - its dominance evaporated. The thing we'd been calling our number-one signal collapsed to roughly noise. "Bullpens win games" turned out to be an artifact of the leak, not a truth about baseball.
What we did about it
We didn't patch one feature and move on. If our most important input had a leak, others might too. So we swept the entire feature set the same way and rebuilt the model from a clean foundation, with every input constrained to information genuinely available at prediction time.
This week, that clean model went live.
I want to be honest about what the swap did and didn't buy us. It did not make the model dramatically more accurate. A de-leaked model is, almost by definition, a little less impressive on paper than the leaky one it replaces, because the leaky one was getting help. What it bought us is something I care about a great deal more: the model's reasoning is now true.
When you open a pick today and look at why the model landed where it did, the drivers you see are real. With the leak gone, signals that had been hiding underneath it surfaced; team-strength measures like Elo and Pythagorean win expectation, the unglamorous fundamentals that actually move games. Before, every pick told you a story about the bullpen. Now it tells you a story you can trust.
The honesty tax on the number itself
The leak left one more fingerprint, and cleaning it up forced a second correction.
A model that's quietly being fed part of the answer doesn't just look more accurate, it becomes overconfident. It learns to speak in certainties it hasn't earned. Ours had drifted there: it would put a number like 75% on a game where the honest figure was much closer to a coin flip. That is the precise opposite of what this platform is supposed to do.
So alongside the rebuild, we recalibrated the model's probabilities - the careful work of making sure that when the model says 58%, it means 58%, and not 58%-that-is-really-53%. The credible interval you see on every pick is only worth anything if the number at its center is honest. Single baseball games are high-variance by their nature - even a clear favorite is closer to a coin flip than fans like to admit - and the recalibrated model now says so plainly, instead of dressing a near-toss-up in confidence it can't support.
That is not a fun thing to ship. "We made our numbers less confident" is not a growth-hack headline. But a confident wrong number is worse than an honest certain one, and we would rather show you the honest one every single time.
Why I'm writing this down
I could have fixed all of this quietly. The leak was internal; no user would have ever known. That is precisely why I wanted to write it down.
The whole premise of Credence Sports is that you shouldn't have to take a model on faith, that we show our work, including the parts that are unflattering. A platform that only tells you about its wins is asking for the same blind trust as the touts it claims to be better than. The real test of whether "transparency" means anything is whether it survives contact with a mistake. This was ours. We found it, we fixed it, and now you know about it.
The model is better today than it was a week ago. Not because it's flashier - it's quieter, less certain, less eager to impress. It's better because it is more honest about what it knows and what it doesn't. That trade, impressiveness for honesty, is the one we will make every time.
We operate in the penumbra: the space between certainty and ignorance where careful reasoning has an edge. Careful reasoning includes being willing to find out you were wrong. This week we were, and we're better for it.
-- Charlie