Backtesting Discipline: The Easiest Number to Fake
The determining question is when the strategy logic was fixed relative to the performance period being shown.
Overfitting Has a Familiar Shape
A strategy backtest iterated against the same historical dataset dozens of times will eventually appear excellent, not because the underlying logic is sound, but because sufficient parameters have been tuned to fit the noise in that specific data. The resulting performance statistics reflect the fitting process as much as the strategy itself.
“A strategy backtest iterated against the same historical dataset dozens of times will eventually appear excellent, regardless of whether the underlying logic is sound.”
What Out-of-Sample Discipline Actually Means
Genuine out-of-sample discipline means locking the strategy logic before testing it against data the model has never encountered, and discarding a strategy that performs well in-sample but fails out-of-sample.
The Test That Matters More Than the Sharpe Ratio
The test that carries more weight than any single performance statistic is procedural: can the team demonstrate precisely when the strategy logic was locked, and show performance on data that postdates that lock. A provider unable or unwilling to demonstrate this distinction is asking an institution to trust a number that may describe only the fitting process.
- Lock strategy logic before testing against out-of-sample data, resisting the impulse to re-tune after a disappointing result
- Demand a clear record of when strategy logic was finalized relative to the performance period being shown
- Weight the discipline of the testing process at least as heavily as the resulting performance statistic
Discuss this with our team.
Tell us what you are working through, and we will route you to the right partner.
Questions Before Licensing Systematic Trading Technology
The consequential questions concern system behavior under stress, not the marketing claims made about it.
Systematic Portfolios: Diversification That Actually Works
Ten strategies deployed in parallel constitute diversification only if they fail to lose capital in the same week.
Algorithmic Trading Compliance Amid Shifting Markets
A compliance framework calibrated to the prior market cycle governs a market structure that has already moved on.