Index Investing is Misunderstood

The goal is to optimize, not to average

Index investing means trying to invest in a portfolio whose performance will mirror that of a broad stock index, such as the S&P 500. The theory of index investing originated among academic economists working in the field of finance. Index investing slowly caught on with the money management industry and with the public, to the point where it is often used and even misused*. But most people do not understand the theory.

I will try to explain the theory. I will warn you that it is counterintuitive, because it says that you are not rewarded for taking risk that you could have diversified away. Your intuition tells you that if you are willing to take more risk, then buying a single stock is a better idea than buying a diversified portfolio. The theory says that you are wrong. A student taking a class in finance from Fischer Black at MIT once insisted that you would get a higher expected return by buying a risky stock. In utter frustration, Black silently turned around and wrote

NO

on the blackboard in the largest letters possible.

I expect a lot of your comments to show an equal inability to grasp the theory. Please have the humility to say “I do not understand. . .” rather than “Kling is clearly wrong…”

The Portfolio Separation Theorem

The theory of indexing starts with the assumption that an investor cares about two characteristics of the expected future returns on a portfolio: mean (expected future returns, on average), which you want to be high; and variance (a measure of risk), which you prefer to be low.

If you care about mean and variance in this way, then exactly two portfolios are needed in order to optimize. This is known as the portfolio separation theorem.

The portfolio separation theorem is a pain to prove. But here is some intuition for it. Suppose you start with two portfolios, A and B, each consisting of one security, a and b, respectively. Suppose that A has lower mean and lower variance than B. Provisionally, A and B are the optimum portfolios. People who want more mean and are willing to tolerate higher variance should pick more B and less A, but these are the only two portfolios.

Should you add a new security, x to your holdings? If adding some x merely increases expected variance without increasing mean, then the answer is no. We say that x is dominated by the other two portfolios. But if you get the best combination of mean and variance by including some x, then modify A and B so that A now includes both a and x and B includes both b and x. Then repeat this process with new security y, new security z, and so on, until you can no longer increase the mean without increasing the variance.

At each step in the process of forming the optimum portfolio, there are always two sub-portfolios, A and B. The reason that there are only two is that by assumption we only care about two factors, mean and variance. So the optimum is always a weighted average of just two portfolios.

The Right Way to Dial Up Risk

To get from the portfolio separation theorem to indexing, assume that there is a riskless asset, meaning an asset with zero variance. Let a be this asset (perhaps this would be a money market fund that makes very short-term loans in which Treasury securities serve as collateral). Then as we use the portfolio separation process to build the optimal portfolio, A will always consist of the money market fund, and B will consist of every risky security that is not dominated by the combination of other risky securities. In other words, B will be a broad market index.

An investor who wants minimal risk will choose a portfolio consisting mostly of A (the money market fund) with little or no B (the index fund). An investor who can tolerate more risk in hope of getting a high mean return will choose a portfolio consisting mostly of B with little A.

The key lesson of the theory is that the highest possible mean return comes from choosing a lot of the B (the index fund) with little A (the money market fund). It does not come from abandoning B and instead buying just a few stocks.

Most people, including many who embrace index investing, do not buy this lesson. They believe that the more you diversify, the more likely it is that you will get only an average return, not a high one.

I should note that if you truly know something about a stock that other people don’t know, then you can earn an extraordinary return by betting on that stock. Let us set that aside and assume no particular inside knowledge.

I think that most people suffer from what I might call the “mental accounting” fallacy. In terms of mental accounting, you might say that you have $50,000 to allocate to stocks. With only $50,000 to allocate, buying one stock can give you higher return and higher risk than buying $50,000 of an index fund.

What the theory of index investing would say is that you can do better than buying $50,000 of one stock by instead buying more than $50,000 of an index fund. If you buy, say, $60,000 of an index fund, then you will get higher return at less risk than buying $50,000 of one stock.

You might say, “I only have $50,000 in my brokerage account. So I cannot buy $60,000 of an index fund.” But you can. You can buy out-of-the money call options on the S&P 500 index that give you the equivalent of owning $60,000 in the index while putting up much less money.

The theory says that to reach for a higher return, you should take a more leveraged position in the index. You can do that using options. Do not let the mental accounting fallacy get in the way.

Note that I am not saying that you should want to take a leveraged position in the S&P 500. What I am saying is that if you do want to take a high-risk, high-return position, you should prefer a leveraged position in a broad index to a narrow gamble on just one stock.

Any time you read something that says indexing is a way to get the average return on the stock market, realize that this is a basic misunderstanding of the theory of indexing. You understand the theory of indexing if you understand that the market index is the optimum mix of stocks for any investor, regardless of their willingness to take risk, provided that the investor only cares about mean and variance and has no inside knowledge. It says that you can change your risk-return profile by changing the weights that you give to investing in a money market fund and investing in a broad market index fund.

You dial up your expected risk and return by making a larger bet on the index. You dial down your expected risk and return by instead putting more of your assets in the money market fund and refraining from using options to lever up your position in the stock market.

*The most common misuse of indexing is to buy stocks that mirror a particular index, such as the index of energy stocks, rather than a broad index. The theoretical justification for indexing that I tried to explain here does not apply in the case of this sort of “narrow” indexing.

Along these lines, I have to say that I am not sure that these days the S&P 500 is close to the theoretical market index. Its value is heavily influenced by just a few stocks, including Alphabet, Amazon, and Microsoft. Does this really give investors as much diversification as the theory of index investing would call for?

This essay is part of a series on human interdependence.