After decades of experience with trading systems and CTA’s, we are comfortable with the phrase (Past Performance is not necessarily indicative of future results), as well as disclaimers for hypothetical performance due to back testing. We’ll be the first to recognize the use of back testing on top of the mandatory disclaimers due to the back testing. But what irks us, is when other advisors/investors aren’t held to the same standard, pretending that almost every other investment they look at is not back tested… Take this nice quotation from MorningStar’s Samuel Lee:
“In a real sense, any investor who observes a historical pattern has engaged in back-testing.”
To be fair (and we agree), the article goes on to say that true back testing done correctly, requires quite a bit more than just looking at historical patterns. But after reading a recent Barron’s blog post, it seems that back testing (by our definition) isn’t just for managed futures and other alternative investments anymore. It seems that back testing is being applied to the $1.7 trillion ETF market. And even if back testing is done correctly (which is a difficult task to accomplish), there are still some pitfalls.
The issues regarding back testing are real and at times hard to avoid. Nearly all trading strategies including systematic and discretionary traders (both those traded by individual investors and professionals) rely pretty heavily on back tested performance to validate their models and/or trading ideas. The fundamental problem being that who would ever create a trading strategy that didn’t hold up over its initial testing period? MorningStar provides a good example.
“Last year, Vanguard published a study on ETFs tracking back-tested indexes. The authors, Joel M. Dickson, Sachin Padmawar, and Sarah Hammer (1), looked at a sample of equity indexes with at least five years of back-tested history and five years of live performance. In the five years prior to index live date, the indexes averaged 12.25% excess returns above U.S. equity market; five years after live date, they averaged negative 0.26%. In other words, most ETF index back-tests are garbage.”
Of course, the issue with this is that even the most cautious system developer/coder/writer can unintentionally fit the parameters of their system to what the market(s) have done in the past; creating a system that looks great in testing, but in reality has little to no chance of holding up in the future. This is what we in the industry call “Curve Fitting”. And, while our experience has taught us to be cognizant of this bias and ask pertinent questions to flush it out during our due diligence, we wonder if everyday stock investors realize that many of the latest and greatest ETF innovations also have the potential for being curve fit to the market. Take Savita Subramanian and three coauthors of Bank of America Merrill Lynch’s quantitative strategists.
“Given our quantitative roots, we are sympathetic to the fact that back tests are often used as an input into making investment decisions. But past returns, as we all know, do not predict the future. And we think backtested results may be particularly problematic today. Very little fundamental data for US equities extends back more than 30 years, but the last 30 years were a period generally accompanied by two related phenomena: increasingly easy monetary policy and falling interest rates. In particular, the wave of liquidity and stimulus provided in the wake of the Tech Bubble coincided with unprecedented levels of credit expansion, rising asset correlations and record earnings volatility.”
The catch is that advisors and ETF marketers cannot show investors hypothetical performance when promoting a new ETF to the client…but of course, Wall St. being Wall St. you know there is a workaround of sorts. A workaround so obvious that most investors probably won’t even catch it at first…the potential for back test bias is in the actual index the ETF is attempting to mimic. See approximately 90% of all ETF’s are “passive” investments aka they attempt to model an index like the S&P 500 or a single market like Gold. And what Wall St. and ETF creators have figured out is that they create an index to mimic almost any investment strategy… An index that CAN be shown to potential investors for marketing purposes (with the proper disclaimers of course). An index that is…you got it, back tested (emphasis theirs).
“IN THE SEARCH FOR THE NEW and different indexes that will power a new and different ETF, back-testing plays a critical role. Index providers, including S&P, Dow Jones, MSCI, Russell, Zacks and others can index just about anything. You want to rank the companies in the S&P 500 by earnings growth, then take the 50 top firms and weight them equally? Weight them by market capitalization? Go long the top 50 and short the bottom 50? They can build it. And then they back-test it. Indexes that look good in hindsight have a shot to become ETFs. Those that don’t, don’t.
That approach allows fund companies to cherry-pick, in order to launch funds that can appear rosy. The problem with that, says Joel Dickson, senior ETF strategist at Vanguard, is that there’s no guarantee that the parameters and rules that would define an index you would create today would have been the same ones you would have used, say, five or 10 years ago.
“In 2008, you might construct an index that was tilted more toward value factors,” Dickson says. “And the back-test would show terrific outperformance. But in 2001, you would never have tilted toward those value variables in the same way.”
Nor has hypothetical past performance been an indicator of future success. Vanguard’s research shows that the average back-tested index outperformed the market by 10.3% per year in the five years prior to an ETF’s launch, then underperformed by 1% a year in the subsequent five years.
Now the authors quoted here aren’t making any claims of fraudulent activity going on in the ETF creation process. In fact we have had many conversations with smart people on the ETF side who are working on some great, innovative products and genuinely have their clients’ best interests at heart. All the authors/we are saying is that investors need to be aware that the ETF creation process is a form of financial engineering and that just because an ETF is a passive investment doesn’t mean that tough questions don’t need to be asked.