Step 5 of 7

Tail Holdout

The only number in this entire system that was not used to choose anything. The configurations are frozen; the last N bars are set aside; the same simulator is re-run on data the search never saw.

⚠

Read this before acting on any other number Every metric produced by stages 2, 3 and 4 is in-sample. The search selected the configurations that looked best on that exact data — reporting those figures as expected performance is curve-fitting with extra steps. This stage is the first and only place in the pipeline where performance is measured on data that had no influence on the choices.

What this stage does

# run both modes
holdout_tail.py --symbol XAUUSD --tf M5 --folder all9 --mode warm --tail 2500 --cores 14
holdout_tail.py --symbol XAUUSD --tf M5 --folder all9 --mode cold --tail 2500 --cores 14

Read the frozen portfolio

<root>/<folder>/summary.json — the slot parameters, exactly as the search left them. Nothing is re-optimised.

Compute the tail boundary

tail_start = n_bars − tail_bars. With 10,000 bars and a 2,500-bar tail, the tail occupies bars 7,500–10,000.

Re-simulate every slot with the tail hooks set

The same engine, the same feature code, the same parameters — restricted so that no new position opens before tail_start, and (in cold mode) so that no learning happens before it either.

Truncate everything to the tail

Metrics and trades are computed only from bars at or after the boundary. The pre-tail history exists solely to warm the model up.

Sum the slots and write the report

Per-slot and combined metrics for the tail window, plus the equity curve and every tail trade.

Warm vs cold — the whole point

Warm — default

Mirrors live trading

The model learns and re-trains normally from bar 0 — exactly as it would have been doing in the weeks before the tail — but takes no new positions until the boundary.

This is what a real EA looks like on the day you attach it: an already-adapted model meeting fresh prices.

Cold — strictest

From scratch

No learning at all — no online steps, no periodic re-trains — until the boundary. Then one batch train on the pre-tail window (labels ending strictly before the boundary), then normal online learning.

Nothing the model knows was derived from a bar it also traded.

How the two are implemented

Both are plain parameters on the same simulator, not separate code paths. That is what makes the comparison meaningful — you are varying one behaviour, not comparing two implementations.

Mode	Engine hooks
Warm	`_trade_from = tail_start` only. Learning runs from bar 0 as usual.
Cold	`_trade_from = tail_start` and `_cold_start = True`. Learning is suppressed until the boundary.

✓

The gap between them is itself informative If warm is strongly positive and cold is negative, the edge depends on a model that has already adapted to the regime — which means it would need a comparable warm-up period in live trading before it started working. If both are similar, the edge is less dependent on inherited state.

They frequently disagree. That disagreement is a finding, not an error.

Metrics are computed from the tail only

warmup   = training_bars + 1
k0       = tail_start - warmup

tail_eq = equity[k0:]                       # start from the tail's warmup bar
tail_eq = tail_eq - tail_eq[0] + initial_balance   # rebase to the initial balance

tail_trades = trades[trades.bar_entry >= tail_start]

The equity curve is rebased so the tail starts at exactly the initial balance — making the tail's net profit and drawdown independent of whatever the warm-up period happened to do. The warm-up's P&L is deliberately excluded: it was not part of the holdout.

Choosing the tail length

--tail defaults to 2500 bars. At M5 that is roughly 8–9 trading days of continuous bars. Some considerations:

Consideration	Detail
Hard floor	`tail_start` must be at least 1000 bars, or the script exits with "need ≥1000 bars of pre-tail history for warmup".
Warmup requirement	Each slot needs `training_bars + 1` bars before the tail starts. With `training_bars` up to 1500, a 2,500-bar tail on 10,000 bars leaves ample room; a very long tail on a short file does not.
Too short	With 30 trades as the acceptance floor for the search, a tail yielding only a handful of trades cannot distinguish an edge from luck.
Too long	Eats the data the search needed. The slots were chosen on the earlier portion; if the tail is most of the file, the search had very little to work with.

ℹ

Keep the tail length in sync with the config The staleness check compares the tail report's recorded n_bars against the current CSV's row count. If you change the CSV, the holdout is flagged stale — because the frozen configurations were selected on different data and the result no longer describes the instrument you have.

Progress output

all9: 9 slots | data=XAUUSD_M5.csv n=10000 | tail=2500 bars [7500..10000) | mode=warm
  slots simulated in 7s

=== all9 TAIL 2500 bars (mode=warm) ===
  magic= 401 base9       NP=+9  DD=-0.12% T=142 W= 76 B= 70 S= 72  PnL/t=+0.06 | sl=2.0 rr=1.5 la=3 tb=1000 ri=500 blr=0.01 th=0.55
  magic= 411 momentum    NP=+4  DD=-0.21% T=118 W= 61 B= 57 S= 61  PnL/t=+0.03 | ...
  ...
  TOTAL trades=814  BUY=400  SELL=414

COMBINED TAIL: NP=+57  MaxDD=-0.41%  Sharpe=1.23  final=10057 (2500 bars = 8.7 days M5)

wrote strategies_lr_m5/all9/tail2500_warm/  (tail_report.json, equity_combined_tail.csv, trades_<magic>.csv)

Outputs

Under <root>/<folder>/tail<tail>_<mode>/:

File	Contents
`tail_report.json`	`folder`, `mode`, `tail_bars`, `n_bars`, `tail_start`, a `combined` block (`np`, `max_dd_pct`, `sharpe`, `buys`, `sells`, `trades`), and a `slots[]` array of per-slot rows.
`equity_combined_tail.csv`	The combined curve over the tail window.
`trades_<magic>.csv`	Every trade each slot took inside the tail — the raw evidence to inspect by hand.

A per-slot row carries magic, set, np, dd, sharpe, trades, wins, buys, sells, pnl_per_trade, plus the geometry that produced it (sl, rr, la, tb, ri, blr, th).

The Holdout page

Controls

Portfolio folder (all9/all6), tail length, mode, cores.

Three-way comparison

The in-sample headline, the warm tail, and the cold tail — side by side. This is the comparison that decides whether to trade the thing.

Combined tail curve

Equity and drawdown over the holdout window only.

Per-slot table

Which slots survived the tail and which did the heavy lifting.

Trade lists

Direct access to each slot's tail trades.

How to read the result

Warm	Cold	Interpretation
positive	positive	The strongest outcome available here. The edge does not depend on inherited state. Proceed to compile.
positive	negative	The edge depends on an already-adapted model. It would need a warm-up period live before working. Proceed cautiously.
negative	positive	Unusual. Probably a short tail with few trades and a skewed result. Look at the trade count before inferring anything.
negative	negative	No demonstrated edge on unseen data. Stop. Do not compile and do not deploy.
~flat	~flat	No edge, but no loss either. The in-sample result was the search fitting noise.

⚠

Check the trade count before believing any of this A positive tail with eleven trades proves nothing. Ask how many trades the tail produced per slot, and whether that is enough to distinguish an edge from variance. The per-slot table shows it, and the trade CSVs let you look at the individual results.

A single window is not a guarantee

The tail is one contiguous period — one market regime. A book can be positive here and still fail on the next two months, because the next two months may not look like this tail. This stage is a strong filter against gross overfitting; it is not evidence that the strategy will work going forward.

That is why the recommendation throughout these docs is to forward-test on a demo account. The holdout tells you the strategy is not obviously curve-fitted. Only live data tells you whether it still works.

Checkpoint

✓

Before moving to Step 6

Both modes have been run, not just warm.
Both are positive — or you have consciously accepted a warm-only edge.
The tail produced a meaningful number of trades.
The on-disk CSV has not changed since the search ran — otherwise the whole chain is stale.

Symptom	Cause	Fix
`summary.json` missing	The portfolio folder has not been built.	Run Step 3 first, or check `--folder` matches the folder that exists.
`tail too large: need >=1000 bars of pre-tail history`	`tail_start` fell below 1000.	Shorten the tail, or use a longer data file.
WARN: `tail_start < warmup`	A slot's training window is longer than the pre-tail history available.	Shorten the tail. The script falls back to using the whole curve, which weakens the holdout.
Tail shows far fewer trades than the in-sample period	Expected — the tail is a fraction of the data. Compare trades-per-bar, not raw counts.	Nothing to fix, but factor it into how much you trust the result.
Result changed completely after re-running	Non-determinism, or the underlying data or slots changed.	Check whether the CSV or the search output changed; the holdout reads both.

Full troubleshooting catalogue →