Sentiment Analysis

Scope. This article describes what sentiment measures attempt to capture, how they are built, and why they are unusually hard to evaluate. It gives no readings to watch, no levels and no contrarian rules. No backtested result appears and no sentiment measure is described as reliable or unreliable.

It sits alongside the fundamental and technical traditions as a third thing to look at, and its underlying claim is different from either: not that a company is worth something, and not that price history predicts price, but that the state of other people's expectations is itself measurable and informative.

That claim is coherent and the measurement is genuinely difficult, and this article spends most of its length on why — because the difficulty is not usually explained, and it is what determines how much weight the whole field can bear.

The four families of measure

Surveys. Participants are asked what they expect. Bull-and-bear surveys of individual investors and of newsletter writers, institutional expectation surveys, and broader consumer-confidence measures all sit here. They measure what people say.

Positioning and flow. Options activity such as the ratio of puts to calls, short interest, fund flows, futures positioning by category of participant, and margin balances. They measure what people have done.

Price and volatility derived. Implied-volatility indices, the shape of the volatility surface, and credit spreads. These are prices rather than opinions, and they are the most objective of the four — though what they measure is the cost of insurance, which is not the same thing as fear.

Text. Automated reading of news, regulatory filings, earnings-call language and social media, scored for tone.

Why text-based measures are harder than they look

Scoring language for tone requires deciding which words are negative, and financial language is not ordinary language. A study published in the Journal of Finance in 2011 examined a large sample of annual filings and found that almost three-quarters of the words flagged as negative by a widely used general-purpose dictionary are not negative in a financial context. Words such as liability, cost, depreciation and capital carry no negative charge in a financial statement; they are simply the vocabulary of accounting.

The consequence is that a naive tone score of a financial document is substantially measuring how much accounting is in it. Purpose-built financial word lists exist because of this finding, and the episode is a useful general warning: an automated measure can be precise, reproducible and consistently wrong, and none of those first three properties will reveal the fourth.

The contrarian framing, and what it costs

Sentiment is most often read against the crowd — extreme optimism treated as a warning, extreme pessimism as its opposite. The reasoning is not silly: if participants are already fully committed, the marginal buying power that would push prices higher has been spent.

The difficulty is what that framing does to the claim's testability. Every reading becomes a statement about where the current level sits in a distribution, so the claim depends entirely on the definition of extreme — and extreme is defined by percentile of history, which changes as history accumulates. A reading that was in the top decile of its first ten years may be unremarkable across thirty.

The measurement problem, computed

Sentiment series are highly persistent: a bullish week is followed by another bullish week far more often than not. That persistence means consecutive readings are not independent observations, and the effective sample is much smaller than the raw count.

For a series with first-order autocorrelation ρ, the effective number of independent observations is approximately n(1 − ρ)/(1 + ρ). Twenty years of weekly readings is 1,040 raw observations.

The second half of the calculation asks how many independent episodes would be needed to establish that outcomes after extreme readings differ from a coin flip.

Worked example — two decades of data cannot settle the question. A weekly sentiment survey with autocorrelation of 0.8 yields roughly 116 effective observations over twenty years, and about 12 independent extreme episodes. Detecting even a 70% hit rate would require 47 episodes; detecting a 60% hit rate would require 194. With twelve episodes, only an effect of overwhelming size — around 80% — would be detectable at all. This cuts in both directions and that is the point. It means confident claims that a sentiment measure works are not supported by the available data, and equally that confident dismissals are not either. The honest description is that most sentiment claims are, at present sample sizes, not testable within a professional lifetime of data — which is a different and more uncomfortable statement than saying they are wrong. (Effective-sample formula and a two-sided binomial power calculation against a 50% null; both are standard constructions and the figures follow from them exactly.)

What has actually been established

The most influential result in the area is specific and worth stating precisely. A study published in the Journal of Finance in 2007 measured the tone of a daily newspaper column quantitatively and found that high media pessimism predicts downward pressure on prices followed by a reversion toward fundamentals, and that unusually high or unusually low pessimism predicts high trading volume. The author reported these results as consistent with theoretical models of noise and liquidity traders, and as inconsistent with media content being merely a proxy for new information about fundamental values, a proxy for volatility, or a sideshow unrelated to markets.

Two structural problems that no amount of data fixes

Sentiment largely follows returns. People report feeling optimistic after prices have risen. A sentiment series is therefore substantially a lagged transformation of the price series, which places it in the same category as the indicators in the article on moving averages, RSI and MACD — a rearrangement of information already present rather than an addition to it. The burden on any sentiment measure is to show it adds something beyond trailing returns, and that is a much harder demonstration than showing it is correlated with what follows.

And the measure is part of the system it measures. A widely followed sentiment gauge is watched by participants who adjust for it, which changes the behaviour it was built to observe. This is the same reflexivity that closes the technical cluster in Support and Resistance, and it applies with more force here, because sentiment is by definition a measurement of other participants rather than of an asset.