Quantitative Analysis: The Concept and Its Hazards

Scope. This article explains what quantitative analysis is as an approach and what statistical hazards are specific to applying it to markets. It does not teach strategy construction. No factor, rule, model specification or parameter is offered, no backtested result appears, and nothing here is a recipe. It is the counterpart to Pillar 22's article on multiple testing, which states the underlying statistical argument in general form.

A quantitative analyst specifies what will be measured, over what universe, on what data, and then applies that specification everywhere without exception — including to the cases where it produces an uncomfortable answer.

That definition contains the approach's real contribution and its central hazard at the same time. The contribution is that a rule stated in advance can be checked, which a judgement cannot. The hazard is that a rule can be discovered by searching rather than reasoned to, and a rule found by searching looks identical to one that was reasoned to.

The term covers three activities with very different standing

Measurement and risk — computing exposures, volatilities, correlations and concentrations across a portfolio. This is largely descriptive statistics applied carefully, and it is the best-founded part of the field.

Execution and cost — modelling how orders move prices, how much trading costs, and how to schedule it. This is an engineering problem with observable feedback, and practitioners can tell relatively quickly whether they are right.

Return prediction — identifying characteristics associated with future returns. This is the contested part, it is what most people mean by the word, and almost everything difficult in this article is about it.

Conflating the three flatters the last by association with the first two. A firm can be excellent at measuring risk and no better than chance at predicting returns, and the two capabilities have almost nothing to do with each other.

What the approach genuinely does better

Consistency. The same criteria are applied to the four-hundredth company as to the first, without fatigue, boredom or a prior opinion about the management.

Breadth. A rule can examine a universe no analyst could read through.

Auditability. The decision can be reconstructed afterwards. Why a discretionary decision was made is often not recoverable even by the person who made it.

Falsifiability, which is the real one. A rule stated precisely can be shown to be wrong. A judgement expressed as the shares look attractive here cannot be, because it has no content that could fail. The discipline of stating a claim precisely enough to be refuted is the approach's actual intellectual contribution, and it survives every criticism below.

Hazard one: fit arrives without any relationship being present

Adding explanatory variables improves in-sample fit mechanically, whether or not those variables have anything to do with the outcome. The expected in-sample R-squared from fitting k predictors to n observations of pure noise is approximately k divided by (n − 1) — and the fit is entirely spurious by construction.

The table below fits random predictors to a random target over 250 observations, then applies the fitted model to fresh data.

Worked example — a 40% fit from nothing at all. One hundred meaningless predictors on 250 observations of noise produce an in-sample R-squared of 0.40, matching the theoretical value almost exactly. Applied to fresh data from the identical process, the same model produces an R-squared of −0.69 — meaning it is substantially worse than having used the sample mean. The in-sample figure is not evidence that has been overstated; it is not evidence at all. It is a deterministic consequence of the number of parameters relative to the number of observations, and it would appear identically if the predictors were the results of coin flips.

Hazard two: the best of many is not the best, it is the luckiest

This is the multiple-testing problem, and in a quantitative setting it operates at industrial scale. If N candidate strategies are evaluated and the best is selected, the test statistic of the winner is the maximum of N draws, and the maximum of many draws is large even when every individual draw is meaningless.

Worked example — where the conventional significance bar goes to die. Search 1,000 candidates on data with no structure in it and the best one clears the conventional t = 2.0 threshold in 100% of runs, with an expected t-statistic of 3.24. The convergence with the published literature is the striking part. A 2016 study in the Review of Financial Studies catalogued 316 factors proposed to explain the cross-section of returns, built a multiple-testing framework around that search, and concluded that a newly proposed factor should have to clear a t-statistic above 3.0 rather than the customary 2.0 — and that most claimed findings in financial economics are likely to be false. An arbitrary simulation of a thousand noise trials lands at 3.24; a careful accounting of the actual published search lands at 3.0. They are measuring the same thing from opposite directions. (Simulated values; the expected maximum of N standard normal draws is a known quantity and the figures reproduce it to the second decimal.)

Hazard three: the record is far shorter than it looks

A track record is a sample, and its length governs what can be concluded from it. The standard error of an annualised Sharpe ratio is approximately the reciprocal of the square root of the number of years observed.

Hazard four: the subject changes when it is studied

A physical constant does not react to being measured. A market does. When a characteristic associated with returns is published, capital moves toward it, and the movement of capital is precisely what would remove the effect. The literature documents this decay after publication, which means a finding can be entirely genuine at the time of discovery and worthless by the time it is widely known — without anyone having done anything wrong.

The consequence for evaluating any quantitative claim is severe. An out-of-sample test on data from before publication is not really out of sample in the sense that matters, because the world had not yet adapted. The only fully honest test runs forward from the moment the claim was made, and that test takes as long as the horizon of the claim.

Hazard five: the data is not what it appears to be

Survivorship. A universe assembled today contains the companies that still exist. Testing a rule on it asks how the rule would have done on the survivors, which is not a question anyone needed answered.

Look-ahead. Financial statement data is dated to the period it describes, not the date it became public. Using it on the earlier date embeds knowledge nobody had.

Restatement. Databases hold the current version of a figure, not the version originally published. A rule tested on restated figures was tested on information that did not exist at the time.

Point-in-time data solves all three and is expensive, which is why it is frequently not used.

The honest position

Quantitative analysis is not a superior method or an inferior one; it is a method whose failures are legible. A discretionary analyst who has been wrong for ten years can usually explain why the ten years were unrepresentative. A quantitative claim written down in advance cannot. That legibility is why the criticisms in this article exist in such detail — the field generated them itself, about itself, in its own journals. No comparable literature exists auditing the reliability of judgement, not because judgement is more reliable but because it is harder to hold still long enough to be examined.