Walk-forward testing vs backtesting

The two questions people call 'a backtest'

Backtests can answer different questions even when they start with the same trading rule. State which question a report answers before comparing its results with another report.

A descriptive backtest simulates fixed rules over a chosen history. It can help check the implementation. Its performance evidence depends on how the rules were selected and how much of that history influenced the selection.

The second question concerns a decision process: what would a rule chosen on earlier history have produced during a later test window? Walk-forward testing simulates that process. It can still fail when the market regime, data or execution assumptions change.

How a walk-forward run is built

Construct chronological training and test windows. Fit within each training window, evaluate the next held-out window, and retain its trades and assumptions. Stitch the held-out observations chronologically, taking care not to count overlapping test periods twice.

1. Split. Cut the history into ordered train/test pairs. A random split can put later observations into training and earlier observations into testing, so it does not preserve the sequence of a trading decision.
2. Purge. Remove training observations whose labels overlap the test window. A forward-return label can include prices beyond its observation date, so splitting rows by date alone may leave leakage.
3. Embargo. Restrict adjacent observations where the validation construction could carry test information into a training set. For a chronological walk-forward, document the train-to-test gap and label horizon. The TradePolaris methodology describes a buffer of one trading week.
4. Fit, then trade. Fit only on the training window. Trade the test window with no refitting inside it.
5. Stitch. Concatenate the test-window trades chronologically. The result is one series with no in-sample observations in it at all.

State execution assumptions beside the simulated results. Signals fill at the next bar's open. Stops that gap through fill at the gap price. Costs are charged per side. Retain these conventions so a reviewer can inspect how the simulation turns a signal into a trade.

Why fitted and held-out results differ

A difference between fitted and held-out results can reflect selection, sampling variation, transaction costs or a change in market conditions. Record those explanations separately. A performance gap alone does not measure selection bias.

There is no universal acceptable ratio between fitted and held-out Sharpe. Review the fold results, estimation uncertainty and trial history before deciding what a gap means for a particular experiment.

Write down the evaluation rule before inspecting the held-out result. If the result changes the hypothesis, record a new experiment and include it in the trial history.

What still goes wrong in a correctly built walk-forward

The universe was chosen with hindsight

Applying today's index membership to earlier dates introduces a separate source of bias even when the train and test windows are separated. Check membership as it existed at each decision date. TradePolaris grades universe integrity separately. See point-in-time data and survivorship bias for the mechanics.

Repeated selection across held-out results

Changing a feature after inspecting held-out results makes those results part of the selection process. Record the additional trial and reserve fresh evaluation data where possible. A selection correction describes part of that search uncertainty. See the deflated Sharpe ratio and PBO for the assumptions it requires.

Costs were assumed rather than stressed

Recompute simulated trades under documented higher-cost assumptions. A result that changes materially under those assumptions needs further review. TradePolaris reports cost-sensitivity gates as diagnostics, so the reader can examine this sensitivity beside the other verdicts.

One hot streak carries the whole record

A small part of a return history can dominate the aggregate Sharpe. Examine rolling trade windows and individual folds to see how widely the result is distributed. Stability checks describe the historical sample and cannot guarantee persistence.

How to review a walk-forward result

• Inspect the fold table alongside the equity curve. Per-fold held-out returns help show whether a result is spread across periods or concentrated in one.
• Ask how boundary leakage was handled. Inspect label overlap, feature construction and any purge or embargo. Missing documentation leaves uncertainty about leakage. The required buffer depends on how the data and labels were constructed.
• Ask how many variants were run. Inspect the trial count used in Deflated Sharpe and any missing or dependent variants. An incomplete trial record limits interpretation of the correction.
• Ask how the universe was constructed. Read the historical membership support and any current-constituent fallback before interpreting the result.
• Ask what fills at what price. Next open beats same close. Anything better than next open needs an explanation.
• Ask what happened in the weakest fold. Use its exposures and assumptions to understand a possible failure mode.

How this runs in TradePolaris

The TradePolaris full-tier backtest uses walk-forward construction and grades the held-out series with a panel of statistical gates. The report records each verdict so a reviewer can inspect the reason a run qualifies or fails. Read the run's evidence and supported execution path when interpreting it.

Each run keeps the fold records, the gate verdicts and the dataset version it read, so the same evidence can be reopened months later and re-argued. That record is what an investment committee actually asks for. It makes the number defensible and keeps the defence on file, which is why a walk-forward is worth the extra compute.