Skip navigation

Mutual Fund News

Why 'this strategy was back-tested' are words to beware

Avner Mandelman is a director of Venator Capital Management and author of The Sleuth Investor.

One of the problems in setting up an investment fund is that prospective clients want to be reassured by the fund's long record of success - and that is the exact thing that new funds lack.

That's why some aspiring money managers try to circumvent the process by providing a "back-test" of their investment strategy. The idea here is to show how the fund's strategy would have performed if it had been operating in the past.

If you are approached by a fund touting a back-tested strategy, here's my advice - run the other way.

Back-testing involves simulating what would have happened in years gone by if the manager had bought and sold stocks (or other securities) based on certain criteria - low price-to-earnings ratios, high cash flow yield, or whatever.

Since historical databases of stock market information are now widely available, anyone with a computer can back-test any investment strategy, then claim the results are proof that their strategy will work because it has worked before - at least in a computer simulation.

What's wrong with this? Quite a lot, actually.

One big issue is future bias. Regular databases contain only stocks now in existence. They omit bankrupt companies (which the back-tester wouldn't have wanted to own) as well as merged or acquired companies (which he might have liked to own). Leaving out these two groups can skew results.

This problem can be eliminated by using a special database that contains all the securities that existed at the beginning of the test period. But another problem remains.

From the back-tester's perch in the present, he can easily pick and choose the factors he should have focused on years ago to get the best results, had he only known the future.

Picking the right variables today to get good performance yesterday is called forecasting the past. The back-tester can always play around with a basket of variables to find the handful that would have led to success in an earlier era. Unfortunately, there are no guarantees that following these variables will also work in the future.

Another problem is that the tester assumes only numerical data are relevant for investment decisions. This is patently false. Real money managers depend on non-numerical data to spot developing problems - the vodka-breath of a CEO at 10 a.m., for instance. How can one back-test for such factors?

A subtler problem is that some back-tested gains are the result of "phantom trades." These are trades that appear possible in a database but would not have taken place in the real world because trading volume was too thin or the bid would have moved the price of the stock upward.

An even more pernicious flaw is that back-testing ignores the human element. If the market plunges, your money manager may be just as scared as the crowd. When it's time to buy, he may not obey his strategy's signals.

For this reason, I'm also skeptical of money managers who haven't gone through a severe market plunge. If another bloodbath like 2008 hits, wouldn't you want to know how your gladiator did in a real battle, rather than in a video simulation of one?

One famous back-tested strategy was the Dogs of the Dow method, which consists of buying the 10 highest yielding stocks in the Dow Jones industrial average. This looked great on paper, but has produced only mediocre results in recent years.

Another example was the Long Term Capital Management hedge fund. A pair of Nobel laureates back-tested its strategy, yet it failed spectacularly in 1998 when it ran into market conditions the back testing hadn't counted on.

So, is back-testing useless? Nearly always. Even good money managers who use back-tested models, like Jeremy Grantham of global investment management firm GMO, state that their true job is to override the model in critical junctures, based on non-numerical factors - gut feel, in other words. This is how they get much of their return. Unless your gut feel is as good as Mr. Grantham's, avoid back-tested strategies until they have a real track record.

© 2007 The Globe and Mail. All rights reserved.

Search Fund News

Advanced Search

Only GlobeinvestorGOLD combines the strength of powerful investing tools with the insight of The Globe and Mail.

Discover a wealth of investment information and and exclusive features.

Free E-Mail Newsletters

  • Morning news headlines
  • Morning business headlines
  • Financial highlights
  • Tech alert
  • Leisure

Sign-up for our free newsletters

Back to top