Why Backtests Lie

Reading-order note and boundary. Numbered #8 and drafted last, closing the pillar, so it can draw on every article before it. Boundary: #7 covers distortions already present in historical data; this article covers what goes wrong when someone tests a strategy against that data. Backtesting as a quantitative technique — how to construct one properly — belongs to Pillar 25. This article is about why the results a reader is shown should be discounted heavily.

Those statements are compatible, and the reason they are compatible is the whole content of this article: the problem is rarely the arithmetic, which is usually correct. The problem is the selection process that determined which backtest you were shown.

The central problem: enough tries guarantee a good result

Search a large enough space of rules and you will find one that performed spectacularly, whether or not anything real is happening. This is not a subtle statistical caveat — it is the dominant effect, and it is large enough to swamp everything else in the article. The mechanism is simple. Any measurement taken from a finite sample carries error. Try one rule and you get one draw from that error. Try a thousand and take the best, and you have selected the most extreme draw from a thousand — which will look impressive even when every rule tested is worthless. The arithmetic is stark. A performance ratio estimated over twenty years of data carries a standard error of roughly 0.22. Take the best of ten completely worthless strategies and its apparent ratio is about 0.30; the best of a hundred, about 0.53; the best of a thousand, about 0.70; the best of ten thousand, about 0.84. For scale: the canonical figure for the market itself in this pillar is 0.31. So a researcher who tries a thousand rules containing no genuine edge whatsoever should expect to produce a headline result more than twice as good as the market — from noise alone, with honest arithmetic, on real data. The same effect in the language of significance testing: run a thousand tests at a one-in-twenty threshold and about fifty will pass by chance; tighten to one-in-a-hundred and about ten still will. The decisive question about any backtest is therefore not "what did it return?" but "how many variations were tried before this one was shown to me?" — and that number is almost never disclosed, is often not recorded, and is sometimes not even known to the person who ran the search.

Four more ways the number inflates

Overfitting. Every adjustable parameter — a threshold, a lookback window, a rebalancing frequency, an exclusion rule — is another opportunity to fit the noise in the sample rather than the signal. A rule with enough parameters can describe any history perfectly and predict nothing, and the tell is fragility: if performance collapses when a parameter moves slightly, the parameter was fitted to noise. An out-of-sample test is the standard defence and a weaker one than it appears, because if the out-of-sample period is examined repeatedly and the rule revised in response, it has quietly become in-sample.

Omitted frictions. Backtests routinely understate what trading costs. Bid-ask spreads, market impact for size, borrowing costs for short positions, and the simple fact that the tested price may not have been transactable all reduce the result — and the damage scales with turnover, which is precisely what elaborate strategies have. At 100% annual turnover and a 0.20% round-trip cost the drag is 0.20 points a year; at 300% turnover, 0.60 points; at 500% turnover with 0.25% costs, 1.25 points. Over twenty years on $10,000, that last figure removes 20.6% of the outcome — and it is a cost the backtest may simply not have modelled.

Inherited data problems. Everything #7 establishes applies here too: a backtest run on today's index constituents, or on restated financials, or on a database that dropped its failures, produces a flattered result no matter how carefully the rule was specified. A perfectly designed test on contaminated data returns a contaminated answer.

Regime dependence and publication selection. Every test covers a period with particular conditions — a rate environment, an inflation regime, a set of crises — and a rule tuned to those conditions may be describing the period rather than the market. And finally: you only ever see the backtests that worked. Failed tests are not published, not marketed, and rarely even retained, so the population of visible backtests is selected on exactly the quantity that makes them unreliable — the same structural distortion #7 identified in fund records, operating one level up.