Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-Of-Sample Performance
A Talk by Marcos López de Prado
A large number of quantitative hedge funds have historically sustained losses. In this study we argue that the backtesting methodology at the core of their strategy selection process may have played a role.
- Most firms and portfolio managers rely on backtests (or historical simulations of performance) to allocate capital to investment strategies.
- After trying only 7 strategy configurations, a researcher is expected to identify at least one 2-year long backtest with an annualized Sharpe ratio of over 1, when the expected out of sample Sharpe ratio is 0.
- If the researcher tries a large enough number of strategy configurations, a backtest can always be fit to any desired performance for a fixed sample length. Thus, there is a minimum backtest length (MinBTL) that should be required for a given number of trials.
- Under memory effects, overfitting leads to systematic losses, not noise.
- Standard statistical techniques designed to prevent regression overfitting, such as hold-out, are inaccurate in the context of backtest evaluation, because they do not control for the number of trials attempted.
- The practical totality of published backtests do not report the number of trials involved, hence backtest results reported in academic and practitioners' publications are likely to be spurious.
Marcos López de Prado is Head of Quantitative Trading & Research at Hess Energy Trading Company, the trading arm of Hess Corporation, a Fortune 100 company.
Before that, Marcos was Head of Global Quantitative Research at Tudor Investment Corporation, where he also led High Frequency Futures Trading and several strategic initiatives. Marcos joined Tudor from PEAK6 Investments, where he was a Partner and ran the Statistical Arbitrage group at the Futures division. Prior to that, he was Head of Quantitative Equity Research at UBS Wealth Management, and a Portfolio Manager at Citadel Investment Group. In addition to his 15+ years of investment management experience, Marcos has received several academic appointments, including Postdoctoral Research Fellow of RCC at Harvard University, Visiting Scholar at Cornell University, and Research Affiliate at Lawrence Berkeley National Laboratory (U.S. Department of Energy’s Office of Science). He holds a Ph.D. in Financial Economics (2003), a second Ph.D. in Mathematical Finance (2011) from Complutense University, is a recipient of the National Award for Excellence in Academic Performance by the Government of Spain (National Valedictorian, 1998) among other awards, and was admitted into American Mensa with a perfect test score.
Marcos is a scientific advisor to Enthought's Python projects (NumPy, SciPy), to quantum computing firm 1QBit, and a member of the editorial board of several academic publications. His research has resulted in three international patent applications, multiple papers listed among the most read in Finance (SSRN), three textbooks, publications in the leading Mathematical Finance journals, etc. Marcos has an Erdös #3 and an Einstein #4 according to the American Mathematical Society.
Learn more about Marcos on his website www.QuantResearch.info
About the Series
The IAQF's Thalesians Seminar Series is a joint effort on the part of the IAQF (www.iaqf.org
) and the Thalesians (www.thalesians.com
). The goal of the series is to provide a forum for the exchange of new ideas and results related to the field of quantitative finance. This goal is accomplished by hosting seminars where leading practitioners and academics present new work, and following the seminars with a reception to facilitate further interaction and discussion. Click here
for information on the IAQF/Thalesian Seminar Series.
IAQF Members: Complimentary by registering through this site
Thalesian Members: $25.00 Click here
Non-Members: $25.00 by registering through this site