Nobody ever lost money in a Spreadsheet

After decades of experience with trading systems and CTA’s, we are comfortable with the phrase (Past Performance is not necessarily indicative of future results), as well as disclaimers for hypothetical performance due to back testing. We’ll be the first to recognize the use of back testing on top of the mandatory disclaimers due to the back testing. But what irks us, is when other advisors/investors aren’t held to the same standard, pretending that almost every other investment they look at is not back tested… Take this nice quotation from MorningStar’s Samuel Lee:

“In a real sense, any investor who observes a historical pattern has engaged in back-testing.”

To be fair (and we agree), the article goes on to say that true back testing done correctly, requires quite a bit more than just looking at historical patterns.  But after reading a recent Barron’s blog post, it seems that back testing (by our definition) isn’t just for managed futures and other alternative investments anymore.  It seems that back testing is being applied to the $1.7 trillion ETF market. And even if back testing is done correctly (which is a difficult task to accomplish), there are still some pitfalls.

The issues regarding back testing are real and at times hard to avoid. Nearly all trading strategies including systematic and discretionary traders (both those traded by individual investors and professionals) rely pretty heavily on back tested performance to validate their models and/or trading ideas.  The fundamental problem being that who would ever create a trading strategy that didn’t hold up over its initial testing period? MorningStar provides a good example.

“Last year, Vanguard published a study on ETFs tracking back-tested indexes. The authors, Joel M. Dickson, Sachin Padmawar, and Sarah Hammer (1), looked at a sample of equity indexes with at least five years of back-tested history and five years of live performance. In the five years prior to index live date, the indexes averaged 12.25% excess returns above U.S. equity market; five years after live date, they averaged negative 0.26%. In other words, most ETF index back-tests are garbage.”

Of course, the issue with this is that even the most cautious system developer/coder/writer can unintentionally fit the parameters of their system to what the market(s) have done in the past; creating a system that looks great in testing, but in reality has little to no chance of holding up in the future. This is what we in the industry call “Curve Fitting”. And, while our experience has taught us to be cognizant of this bias and ask pertinent questions to flush it out during our due diligence,  we wonder if everyday stock investors realize that many of the latest and greatest ETF innovations also have the potential for being curve fit to the market. Take Savita Subramanian and three coauthors of Bank of America Merrill Lynch’s quantitative strategists.

“Given our quantitative roots, we are sympathetic to the fact that back tests are often used as an input into making investment decisions. But past returns, as we all know, do not predict the future. And we think backtested results may be particularly problematic today. Very little fundamental data for US equities extends back more than 30 years, but the last 30 years were a period generally accompanied by two related phenomena: increasingly easy monetary policy and falling interest rates. In particular, the wave of liquidity and stimulus provided in the wake of the Tech Bubble coincided with unprecedented levels of credit expansion, rising asset correlations and record earnings volatility.”

The catch is that advisors and ETF marketers cannot show investors hypothetical performance when promoting a new ETF to the client…but of course, Wall St. being Wall St. you know there is a workaround of sorts. A workaround so obvious that most investors probably won’t even catch it at first…the potential for back test bias is in the actual index the ETF is attempting to mimic.  See approximately 90% of all ETF’s are “passive” investments aka they attempt to model an index like the S&P 500 or a single market like Gold. And what Wall St. and ETF creators have figured out is that they create an index to mimic almost any investment strategy… An index that CAN be shown to potential investors for marketing purposes (with the proper disclaimers of course).  An index that is…you got it, back tested (emphasis theirs).

IN THE SEARCH FOR THE NEW and different indexes that will power a new and different ETF, back-testing plays a critical role. Index providers, including S&P, Dow Jones, MSCI, Russell, Zacks and others can index just about anything. You want to rank the companies in the S&P 500 by earnings growth, then take the 50 top firms and weight them equally? Weight them by market capitalization? Go long the top 50 and short the bottom 50? They can build it. And then they back-test it. Indexes that look good in hindsight have a shot to become ETFs. Those that don’t, don’t.

That approach allows fund companies to cherry-pick, in order to launch funds that can appear rosy. The problem with that, says Joel Dickson, senior ETF strategist at Vanguard, is that there’s no guarantee that the parameters and rules that would define an index you would create today would have been the same ones you would have used, say, five or 10 years ago.

“In 2008, you might construct an index that was tilted more toward value factors,” Dickson says. “And the back-test would show terrific outperformance. But in 2001, you would never have tilted toward those value variables in the same way.”

Nor has hypothetical past performance been an indicator of future success. Vanguard’s research shows that the average back-tested index outperformed the market by 10.3% per year in the five years prior to an ETF’s launch, then underperformed by 1% a year in the subsequent five years.

Now the authors quoted here aren’t making any claims of fraudulent activity going on in the ETF creation process. In fact we have had many conversations with smart people on the ETF side who are working on some great, innovative products and genuinely have their clients’ best interests at heart.  All the authors/we are saying is that investors need to be aware that the ETF creation process is a form of financial engineering and that just because an ETF is a passive investment doesn’t mean that tough questions don’t need to be asked.


    The scenario goes something like this:
    Newly emerging CTA with stellar performance and non-correlation.
    1. As AUM grows CTA finds that slippage increases in smaller commodities markets. Market allocations are revised on a case-by-case determination. Future performance including correlations incrementally drift from the prior “track record” and back testing process.
    2. Institutional investors shortly become the 800 pound gorilla. Large gorillas do not like volatility. CTA modifies leverage selectively on volatile markets. Performance decelerates away from the prior track record and back testing.
    3. Performance compresses because of migration from volatile markets and reduction of leverage. CTA does the math and finds that 2% management fee on $250 million is more than 20% performance fee on $50 million at a decompressed ROI 15 % performance. CTA focuses harder on volatility control.
    4. Future performance is further divorced from prior track record.

  2. David Healy says:

    When did the ETF market grow to $7.8 trillion (2nd paragraph)? I feel like Rip van Winkle – I must have been sleeping.


  1. […] Just because it is an ETF doesn’t mean it shouldn’t get significant scrutiny.  (Attain Capital) […]

  2. […] The Attain Capital blog had a good post up last month on the “hard issues” facing anyone dealing with […]

  3. […] Here is a good article on some backtesting pitfalls as they apply to ETFs. […]

Speak Your Mind


Interested in distributing or reprinting this content? Check out our reprint policy here.


Forex trading, commodity trading, managed futures, and other alternative investments are complex and carry a risk of substantial losses. As such, they are not suitable for all investors.

The entries on this blog are intended to further subscribers understanding, education, and – at times- enjoyment of the world of alternative investments through managed futures, trading systems, and managed forex, and is not intended as investment advice, or an offer or solicitation for the purchase or sale of any financial instrument. Unless distinctly noted otherwise, the data and graphs included herein are intended to be mere examples and exhibits of the topic discussed, are for educational and illustrative purposes only, and do not represent trading in actual accounts. Opinions expressed are that of the author.

*The mention of specific asset class performance (i.e. +3.2%, -4.6%) is based on the noted source index (i.e. Newedge CTA Index, S&P 500 Index, etc.), and investors should take care to understand that any index performance is for the constituents of that index only, and does not represent the entire universe of possible investments within that asset class. And further, that there can be limitations and biases to indices such as survivorship and self reporting biases, and instant history.

The mention of general asset class performance (i.e. managed futures did well, stocks were down, bonds were up) is based on Attain’s direct experience in those asset classes, estimates of performance of dozens of CTAs followed by Attain, and averaging of various indices designed to track said asset classes.

It should be noted that past market performance is not indicative of future market movement.No market data or other information is warranted by Attain Capital Management as to completeness or accuracy, express or implied, and is subject to change without notice.

Managed Futures Disclaimer:

Past Performance is Not Necessarily Indicative of Future Results. The regulations of the CFTC require that prospective clients of a managed futures program (CTA) receive a disclosure document when they are solicited to enter into an agreement whereby the CTA will direct or guide the client’s commodity interest trading and that certain risk factors be highlighted. The disclosure document contains a complete description of the principal risk factors and each fee to be charged to your account by the CTA.