Backtesting: Validate Before You Trade
Backtesting is the discipline of testing a trading rule against historical data before trusting it with money: replay the past, count every trade the rule would have taken, and read the arithmetic honestly. It is the difference between believing a method works and knowing what it did, how often, and at what cost. The quant lesson supplied the four numbers; backtesting supplies the evidence behind them. A rule tested on one flattering year is an anecdote. A rule tested on every instance it would have flagged, across years that include ugly markets, with every cost counted, is a measurement. The discipline sounds mechanical, and the mechanics are simple. The hard part is honesty: the same history that validates a rule can be mined until it flatters almost anything, and most of backtesting is defense against fooling yourself with your own data.
What a Test Actually Does
A backtest is a replay with rules. You define the entry, the exit, the position size, and the costs, then walk the data forward one bar at a time and record every trade exactly as the rule would have taken it. No hindsight adjustments, no skipping the trades that look bad now, no widening the definition when the results disappoint. The output is a trade list, and the trade list is the truth: win rate, average win, average loss, expectancy, the longest losing streak, the deepest drawdown. From those, the practical questions answer themselves. How much does this method make per unit of risk? How painful is the ride? What is the worst stretch it has survived? A method that cannot survive its own replay at least deserves a smaller account than the one you were about to give it.
The replay must be honest about costs, because costs are where paper edges go to die. Every trade in the test should carry its commission, its spread, and an estimate of slippage, because the historical bar you traded in the replay is a fiction: the real fill would have been worse. A rule that shows 30 dollars of expectancy before costs and 12 dollars of cost per trade is a different business than the raw number suggests. Professional test desks treat the cost model with more suspicion than the signal, and retail testers should copy that habit. The cheapest way to double a backtest's profit is to forget the costs, which is why forgetting them is the most common error in the discipline.
The Split That Keeps You Honest
The core defense against fooling yourself is the sample split. Divide the history into two parts: the in-sample portion, where you develop and tune the rule, and the out-of-sample portion, which you lock away and touch only once, at the end, to see whether the rule survived contact with data it was never fitted to. A rule that shines in-sample and collapses out-of-sample was not a rule; it was a description of the past dressed up as a prediction. The degradation between the two samples is the honest measure of what you actually have. Every tuned method gets weaker out of sample, because tuning always captures some noise along with the signal. The question is how much survives: a small haircut means the edge is real, a collapse means there was never an edge at all.
| Practice | Honest test | Self-deceived test |
|---|---|---|
| Sample | Split: tune in-sample, verify out-of-sample | One flattering window, tuned repeatedly |
| Costs | Commission, spread, slippage on every trade | Raw fills, costs forgotten |
| Rules | Fixed before the test runs | Adjusted until the curve looks right |
| Data | Every signal the rule generates | Cherry-picked instruments and dates |
A Worked Example: One Rule, Two Samples
Take one concrete rule and test it properly. The setup: buy a broad index ETF after two consecutive down days, exit after one up day, risk a fixed 100 dollars per trade. The in-sample window: two years, 2021 and 2022, every signal taken, no exceptions. The rule fires 46 times. Results: 28 winners, a 61 percent win rate, average win 220 dollars, average loss 130 dollars. Expectancy: 0.61 times 220 is 134, 0.39 times 130 is 51, so roughly 83 dollars per trade, about 3,840 dollars across the sample before costs. Subtract 12 dollars per trade of costs and the honest figure is near 71 dollars per trade. So far so good: a real, positive, cost-surviving edge on two years of data, and the temptation at this point is to fund it.
Now the split does its work. Lock 2023 away as out-of-sample, run the identical rule untouched, and read the damage: 23 trades, 11 winners, a 48 percent win rate, average win 205, average loss 140. Expectancy: 0.48 times 205 is 98, 0.52 times 140 is 73, so 25 dollars per trade, and after the same 12 dollars of costs, barely 13. The edge did not vanish; it deflated. In-sample it printed 71 after costs, out-of-sample 13, and that ratio, roughly the tuned edge shedding most of its shine on untouched data, is the normal shape of this exercise. The honest conclusion is not that the rule failed. It is that the rule's true worth is the out-of-sample number, sized accordingly, with the in-sample figure recognized as the flattering draft. A trader who sized the position off 71 dollars per trade would have run roughly five times more risk than the evidence supported.
The third lesson of the example is what the test forbids. The 2023 number invites tuning: shift the entry, loosen the exit, filter the days, and the out-of-sample curve improves. But every adjustment made after looking at the out-of-sample result burns that data, turning it into a second in-sample set, and the tester is back to believing a description. The discipline is to accept the deflated number, or go find genuinely fresh data, and never to recycle the verification set into the workshop.
The Traps That Eat Backtests
Three traps account for most false positives. Overfitting is the first: a rule with many adjustable parameters, tested and retuned until the historical curve shines, has learned the past's noise, and noise does not repeat. The tell is complexity: five filters and three exit conditions usually mean the data was bullied into agreement. Survivorship bias is the second: test a stock strategy on today's index members and the losers that were removed along the way never appear, inflating every result. Look-ahead bias is the third, the quietest: using data in the replay that the rule could not have known at the decision moment, a quarterly figure published weeks later, a daily high that was not yet printed when the entry triggered. Each trap inflates the past, and all three share one cure: assume your test is flattering until you have actively tried to break it.
Run well, backtesting changes the trader more than the method. It replaces the memory of best cases with a full ledger, it prices the costs that intuition skips, and it converts the question "does this work" into "how much did this make, per trade, after costs, on data it never saw." That last phrasing is the discipline itself, and it is the standard a method must meet before an account is risked on it.
A validated edge still has to be assigned to a style: some edges fade, some edges run. The next lesson weighs the two great styles, mean reversion and trend following, by their measured numbers rather than their reputations.
Backtesting Questions
How much historical data does a fair backtest need?
Enough to include at least one regime the rule was not designed for, plus enough trades for the expectancy arithmetic to mean anything. A useful floor is several hundred signals across at least three to five years, split into in-sample and out-of-sample portions. Thirty trades in one calm year measures luck, not method, however clean the equity curve looks.
Why did my backtested edge shrink out of sample?
Because tuning always captures noise along with signal, and noise does not repeat. The in-sample result is the flattered draft, the out-of-sample result is the honest one, and the gap between them is normal. The decisive question is what survives: a rule that still clears its costs out of sample is a real, if smaller, edge, and it should be sized on that deflated number rather than the in-sample one.
Can a backtest guarantee future results?
No, and a good test never claims to. It measures how a fixed rule behaved under past conditions, which is evidence, not promise. Regimes change, edges decay as participants adapt, and the future will contain combinations the record never showed. The test's real gift is a defensible baseline and early warning: when live results drift far below the tested ones for a sustained stretch, the edge is announcing its retirement.
What is the single most common backtesting mistake?
Forgetting costs, in one form or another. Raw fills, zero slippage, and no commission can turn a marginal method into a paper fortune. The second most common is tuning against the verification data until every sample looks like an in-sample set. Both errors share a root: letting the result you want edit the test that was supposed to challenge it.