Backtesting: The Easiest Number to Fake
Any strategy looks good if it is tuned against the same history often enough. What matters is when the rules were fixed.
How a backtest gets inflated
A backtest applies trading rules to past data and reports what would have happened. The weakness is that rules can be adjusted after seeing the result, in a loop of changing a parameter, rerunning and comparing.
Do this often enough and the rules will fit the particular quirks of that dataset. Performance statistics such as the Sharpe ratio rise, although they now describe the tuning process as much as any real edge, which researchers call overfitting. This article is informational and is not investment advice.
Signs of an overfitted result
- Many parameters, each set to an oddly precise value.
- Performance that collapses when a parameter is nudged slightly.
- Excellent results in the test period and no record afterwards.
- A team that cannot say how many versions it tried.
- No account of strategies that were dropped.
What out-of-sample testing requires
The remedy is separation. Part of the history is held back and not looked at during design, the rules are fixed, and only then are they run on the held-back data. If results hold up, that is evidence, and if they do not, the strategy is discarded and the data is not reused to rescue it.
Exhibit 1: Three kinds of performance evidence and how much weight each deserves.
| Evidence | How it is produced | Weight |
|---|---|---|
| In-sample backtest | Rules developed and tested on the same data | Low: shows the rules can fit the past |
| Out-of-sample backtest | Rules locked, then run on data held back | Moderate: depends on the hold-back being real |
| Live or paper record after lock | Rules run forward in time after a documented freeze | Highest: the data did not exist when rules were set |
The middle row is only as good as the team's restraint, because if held-back data is peeked at and the rules revised, it has quietly become in-sample.
The procedural test
An outside institution cannot rerun a provider's research, but it can ask about process, and the answers are revealing.
Can you show when the strategy logic was locked, and the performance on data dated after that point?
A provider with sound practice will have a dated record such as a code repository, a research log or a committee minute, and will be able to separate results into before and after. One that cannot, or will not, is asking the institution to rely on a figure that may reflect fitting alone.
Past performance, however carefully validated, does not indicate future results. Validation reduces one specific risk, which is that the record is an artifact of its own construction.
Where this sits in a wider evaluation
Validation method is one of several things to examine before licensing, alongside stress behavior, traceability and commercial structure. The full list is in questions before licensing systematic trading technology.
In one composite engagement, an asset manager stopped relying on a provider's own performance claims and commissioned independent diligence on architecture, auditability and risk controls before agreeing a license. Read the licensing diligence case study.
Questions to put to any provider
- Ask for the date on which the strategy rules were frozen, with evidence.
- Request performance split into before and after that date.
- Ask how many variations were tested before this one was selected.
- Find out which strategies the team rejected, and why.
Ask us how our strategies were validated
Synergy Nexus Quant can walk your team through when strategy logic was locked and how out-of-sample testing was carried out.
Algorithmic Trading Compliance Amid Shifting Markets
The rules on algorithmic trading controls differ between Canada, the United States and Europe. What supervisors want to see demonstrated is much the same.
Risk Budgeting Across a Multi-Strategy Systematic Portfolio
Giving every strategy the same capital feels balanced. In risk terms it rarely is.
The Case for Architectural Diversification Across Regimes
Spreading across asset classes does not help if every strategy needs the same market condition. Diversifying by how strategies decide addresses that.