Start here

The LR Engine

Every stage of the pipeline runs the same simulator. Understanding it once explains the search space, the metrics, the generated MQL5, and why the holdout matters — so it is worth reading carefully.

The one-sentence version

A binary logistic regression on a small vector of price-derived features, re-trained on a rolling window that ends strictly before the current bar, trading one position at a time with ATR-based stop-loss and take-profit levels, entered at the next bar's open.

The whole design exists to answer one question honestly:

The central claim At the moment the model decides to trade bar k, nothing it has learned was derived from any bar after k. No future price, no future label, no re-fitted parameter that saw the outcome.

The label

Training targets are binary, and depend only on the close price lookahead bars ahead:

label(k) = 1.0  if  close[k + lookahead] > close[k]   else   0.0

So the model is not predicting magnitude, or return, or direction over a fixed horizon in a regression sense. It is predicting whether the close will be higher a few bars later. With the stage-1 default lookahead = 3, that is a three-bar-ahead up/down call.

That is also all the model learns. It never sees P&L, never sees the stop-loss, and is never rewarded for a trade that happened to work out. It optimises classification accuracy on the log-loss of a static binary target.

Why there is no lookahead

This is the part worth understanding properly, because it is a handful of index expressions that carry the entire credibility of the results.

Batch training stops before the current bar

k_start = current - lookahead - training_bars + 1
k_end   = current - lookahead + 1          # exclusive

for k in range(max(1, k_start), k_end):
    ...
    labels.append(1.0 if close[k + lookahead] > close[k] else 0.0)

For every training sample k, the largest index the label touches is k + lookahead. Because the loop stops at k_end - 1 = current - lookahead, the largest label index read is current. The label uses close[k + lookahead] where the bar at k + lookahead has already closed by the time we are at current.

Training also explicitly skips samples whose label would reach past the end of the data: if k + lookahead >= n: continue.

Online learning uses a stale feature vector

def online_learn(i):
    if i < p["lookahead"]:
        return
    stale = i - p["lookahead"]        # the bar whose outcome is now known
    x     = valid_feats(stale)
    pr    = predict(x)
    actual = 1.0 if close[i] > close[stale] else 0.0
    error  = pr - actual
    # one SGD step on weights / bias

Each bar contributes exactly one online gradient step, and it is a step on a fully resolved prediction: the features of bar i - lookahead, compared against the close of bar i. By construction the outcome is known. The model never takes a gradient step based on a prediction whose truth is still in the future.

The two are consistent Batch training and online learning both stop at the same boundary: the set of labels derivable from data known at bar current. That consistency is what makes "no lookahead" a property of the system rather than a claim about one function.

The model

Standard logistic regression with an L2 penalty:

Link function
Sigmoid: $\sigma(z) = \dfrac{1}{1 + e^{-z}}$
Decision
z = X·w + b, then the sigmoid
Loss
Binary cross-entropy (log-loss)
Regularisation
L2 with lam = 0.001 — mirrored in the EA as #define LAM 0.001
Optimiser
Full-batch gradient descent
Iterations
max_iter — 600 during search, 2,000 at final verification
Features
Standardised: (x − mean) / std, with std floored to avoid division by zero
Initial state
Weights zeroed at every batch re-train
Under-trained guard
A batch train is skipped entirely if fewer than 100 usable samples are in the window

Before the first successful train, predict() returns exactly 0.5 — deliberately in the dead zone between the entry thresholds, so an untrained model cannot accidentally trade.

Three separate update paths

PathWhenEffect
Initial train Once, at warmup = training_bars + 1 Full batch gradient descent over the first window. Before this, nothing trades.
Periodic re-train Every retrain_interval bars Discards the weights and re-fits from zero on the shifted window. A hard reset rather than a fine-tune.
Online step Every single bar One SGD step on momentum, using the now-resolved prediction from lookahead bars ago.

The online step uses momentum with a decay factor, so repeated corrections in the same direction accumulate — a small amount of adaptive responsiveness between the hard re-trains.

Re-training resets weights to zero Each batch train starts from weights = 0 and re-derives them by gradient descent. This is not warm-started from the previous fit. It is why retrain_interval is a genuinely meaningful hyper-parameter rather than an incremental one.

The search space

Three groups of parameters get grid-searched, in two stages.

Stage 1 — trade geometry (64 combinations)

ParameterValuesMeaning
sl_mult1.5, 2.0, 2.5, 3.0Stop-loss distance as a multiple of ATR(14).
rr1.0, 1.5, 2.0, 3.0Take-profit as a multiple of the stop distance.
lookahead2, 3, 5, 8Label horizon in bars, and the online-learning lag.

4 × 4 × 4 = 64 combinations, each run at max_iter = 600.

Stage 2 — training behaviour (243 per stage-1 winner)

ParameterValuesMeaning
training_bars500, 1000, 1500Length of the rolling training window. Also sets warmup.
retrain_interval250, 500, 1000How often weights are re-fitted from scratch.
batch_lr0.005, 0.01, 0.02Gradient-descent learning rate for batch training.
long_thresh0.50, 0.55, 0.60Probability above which a long is taken.

3 × 3 × 3 × 3 = 81 combinations, run for each of the top 3 diverse stage-1 survivors — 243 runs.

The three stage-1 survivors are deliberately kept diverse: the list is de-duplicated on the whole (sl_mult, rr, lookahead) tuple, so the three are not three near-identical variants of one good point.

Final verification

The stage 1 and stage 2 results are merged, filtered to those that pass the acceptance test, sorted by score, and the top 40 are re-run at the full budget max_iter = 2000. This matters because the search ran at 600 iterations: a point that looked good under-trained may not survive proper convergence, and vice versa.

The acceptance test and the score

score = net_profit / (1 + abs(max_dd_pct) / 100)

ok    = trades >= 30  and  profit_factor > 1.0

A configuration that fails ok is assigned a score of -1e18, which effectively removes it from the ranking. So the search cannot select a high-net configuration that achieved it in twelve trades or with a profit factor below 1.

The score is a net-over-drawdown ratio Dividing by 1 + |DD%|/100 penalises drawdown but not linearly — a −50% max drawdown divides the score by 1.5, not by 2. It is a ranking device, not a risk model.

Slot selection

After verification, survivors are sorted by score and fed to a greedy selector that picks up to 5 slots subject to a correlation constraint:

Slots per set
N_SLOTS = 5
Correlation measure
Pearson correlation of first-differenced equity curves
Threshold
MAX_CORR = 0.85 — a candidate is rejected if correlation with an already-picked slot exceeds this
Minimum overlap
At least 30 overlapping points, and a denominator above 1e-12, or the pair is not compared
Fallback
If the filter cannot fill all slots, remaining slots are filled by best score ignoring correlation

The correlation is computed on the first difference of equity, not the level. Levels are dominated by the shared price trend of the instrument and would make almost everything look correlated; differencing isolates the per-bar behaviour, which is the thing being diversified.

Trade execution

One position at a time per slot. The sequence for a given signal bar i:

Hour filter
If bar i's server-time hour is not in the allowed set, return immediately. This gates entries only — training is unaffected.
Spread filter (optional)
If enabled and the bar's spread exceeds max_spread, skip. Default is use_spread_filter = False, and the search derives max_spread from the median spread of the data file.
Signal
Compute the feature vector and the probability. Above long_thresh → long. Below short_thresh → short. In between → no trade.
Entry price
The next bar's open: entry = open[i + 1]. Not the current close — the decision is made from closed-bar data, and the fill happens at the next opportunity.
Levels
sl_pts = sl_mult × ATR14[i], then tp_pts = sl_pts × rr. Applied symmetrically around entry according to side.

How positions close

On every subsequent bar the position is checked against its levels. The order is significant:

# long
if low[i]  <= sl:  closed = ("SL", sl)
elif high[i] >= tp: closed = ("TP", tp)

# short
if high[i] >= sl:  closed = ("SL", sl)
elif low[i]  <= tp: closed = ("TP", tp)
The stop is always checked first When a single bar's range spans both levels, the simulator assumes the stop was hit. This is the conservative assumption and it is deliberate: intrabar sequencing is unknowable from OHLC data, and assuming the favourable order would inflate every result. It is also why the MQL5 EA — which sees real ticks — can differ slightly from the backtest.

If neither level is touched, the position carries over. Any position still open at the end of the data is closed at the final close with the reason "EOData".

Breakeven and trailing stops exist in the engine but are off by default, matching the generated EAs. The pipeline describes them as "NOBE" builds.

Money and costs

P&L is computed in dollars by a single expression:

pnl = (exit_price - entry_price) × fixed_lot × lot_mult × side

side is +1 for long, −1 for short. fixed_lot is 0.01 — one micro-lot — everywhere in the pipeline. lot_mult converts a price move into money for that instrument:

Symbollot_multMeaning at 0.01 lot
XAUUSD100$1.00 per $1.00 price move
EURUSD100000$0.10 per pip
DowJones301$0.01 per index point

The EURUSD value is not merely a table entry — it is asserted. On startup the engine derives dollars-per-pip for EURUSD at 0.01 lot and raises a SystemExit if it deviates from $0.10 by more than 5%. A units mistake would otherwise inflate or deflate every figure silently.

Spread is not charged on fills

The engine's P&L is gross The simulator does not subtract spread from individual fills. Instead, the search reads the median of the data file's spread column and the reports estimate cost as median_spread × number_of_trades, always showing raw and net-of-spread figures side by side.

The convention is POINT_DOLLARS = 0.01 — dollars per point per 0.01 lot. This is why a zero-filled or missing spread column is treated as a fatal data error: it would make every trade cost $0 and every result look better than reality.

Magic numbers

Every slot has a unique magic number, built from the feature set's base plus a timeframe offset:

magic = LR_MAGIC_BASE[set] + magic_offset + slot_index

LR_MAGIC_BASE[set] = FEATURE_SETS[set]["magic_start"] + 100

With magic_start = 301 for base9, the base is 401 and its five slots are 401–405. The timeframe offsets let several timeframes run on one account simultaneously without collision:

TFOffsetTFOffset
M14000M305000
M22000H16000
M50H47000
M103000D18000
M151000

So base9 on M5 occupies 401–405, while the same set on M15 occupies 1401–1405. The EA scans for its own positions matching on both magic and symbol.

The features

Features are computed from a small derived-primitive library. Each set selects a subset. All divisions are guarded against 1e-10, and the whole set is standardised before use.

PrimitiveDefinition
logret{n}log(close[i] / close[i + n]) — log return over n bars
atr14, atr10, atr20, atr50SMA of true range over that period
atr_ratioatr14 / atr50 — short vs long volatility
vol10, vol_ratio20atr14 / atr10, atr14 / atr20
atr_sma50(atr14 − atr50) / atr50
rng, rng_atrhigh − low, and that divided by atr14
body_ratio / doji|close − open| / (high − low)
upper_wick, lower_wickWick extents relative to the bar range
close_pos(close − low) / (high − low) — where in the range it closed
ma_cross(sma5 − sma20) / atr14
ma_cross9(sma5 − sma9) / atr14
sma_dist20, sma_dist9, sma_dist50(close − smaN) / atr14
rsi14Relative strength index
mom10close − close[i + 10]
ma_alignCount 0–3 of satisfied orderings among sma5 > sma9, sma9 > sma20, sma20 > sma50
ma_align_dist(sma5 − sma50) / atr14
hh50, ll50Distance to the 50-bar high / low, in ATR units
rng50_atr50-bar high-low range divided by atr14
open, coRaw open, and close − open
hour, dowServer-time hour of day and day of week
Two conventions that surprise people ATR is an SMA of true range, not Wilder smoothing — despite the comments in places. RSI uses a simple average of gains and losses, not Wilder smoothing either. Both are consistent between the Python engine and the generated MQL5, which is what matters for reproducibility — but they will not match a chart indicator that uses the classical definitions.

dow follows pandas' Monday=0..Sunday=6 convention. The MQL5 mirror converts from MetaTrader's Sunday=0 with (DayOfWeekOfBar(t[i]) + 6) % 7.

The eleven feature sets

SetAbbrmagic_startLR baseFeatures
base9BAS301401logret5, logret10, atr14, hour, dow, body_ratio, rng, open, co
momentumMOM311411logret3, logret5, logret10, logret20, hour, dow, body_ratio
volatilityVOL321421atr14, atr_ratio, rng_atr, body_ratio, hour, dow, logret5
price_actionPA331431body_ratio, upper_wick, lower_wick, close_pos, rng_atr, hour, dow, logret5
trendTRD341441logret5, logret10, ma_cross, sma_dist20, atr14, rng_atr, hour, dow
full15FUL35145115 features — the union of the momentum, volatility and price-action ideas
microMIC361461logret1, logret3, logret5, body_ratio, rng_atr, hour, dow
voltrendVT371471atr_ratio, vol10, rng_atr, atr_sma50, vol_ratio20, logret5, logret20, hour
meanrevMR381481sma_dist20, sma_dist50, ma_cross9, sma_dist9, rsi14, close_pos, mom10, hour, dow
mafanMAF391491ma_align, ma_align_dist, ma_cross, ma_cross9, sma_dist20, sma_dist50, logret5, logret20, atr14, rng_atr, hour, dow
breakoutBRK401501hh50, ll50, rng50_atr, rng_atr, atr_ratio, logret5, close_pos, body_ratio, hour

The ALL9 portfolio uses the first six sets plus micro, voltrend and meanrev — nine slots from nine distinct feature definitions. ALL6 uses only the first six.

Metrics

All metrics are computed from the equity curve and the closed-trade list:

MetricDefinition
net_profit (NP) equity[-1] − equity[0] — dollars, at 0.01 lot per slot
max_dd_pct (DD) The minimum of (equity − running_peak) / peak, in percent. A negative number.
profit_factor (PF) gross_profit / gross_loss, with the denominator floored at 1e-10
win_rate (WR) wins / trades × 100
trades (T) Count of closed trades, with buys / sells split
sharpe (S) Annualised from per-bar equity returns
score net_profit / (1 + |max_dd_pct| / 100) — the search ranking device

The Sharpe calculation, precisely

rets  = diff(equity) / equity[:-1]
sharpe = mean(rets) / max(std(rets), 1e-10)
         × sqrt(365 × 24 × 60 / n_bars)

The annualisation factor assumes the bars are minute bars spanning a 24/7 calendar: 365 × 24 × 60 minutes in a year, divided by the number of bars. For an M5 series that over-states the number of bars per year, since the n used is the actual bar count rather than a trading calendar. Treat the Sharpe figure as comparable between slots in the same run rather than as an absolute annualised Sharpe ratio.

Equity starts at the initial balance initial_balance = 10000.0, and the equity array's first entry is that value before any trade. All drawdown figures are therefore relative to a $10,000 notional at 0.01 lot per slot — a very small position size, which is why the dollar figures look modest next to the percentages.

Three modes the engine supports for the holdout

The engine has two hook parameters used only by the tail-holdout stage. They are implemented as plain flags rather than as a separate code path, which is why the holdout is trustworthy: it is the same simulator, restricted.

HookEffect
_trade_from No new entries before this bar. The model still learns and re-trains normally. This is "warm".
_cold_start Suppress all learning — both online steps and periodic re-trains — until _trade_from. Then train once on the pre-tail window and resume online learning normally. This is "cold".

Warm mirrors what a live EA actually does: it has been training and trading before the tail begins, so it arrives at the tail with an already-adapted model. Cold is the stricter test: the model has never seen anything before the tail boundary and receives one batch train from the pre-tail history before trading. See Step 5.

A third mode: signal recording

When record_signals is set, the engine stops simulating trades entirely and instead collects raw entry signals — bar, side, entry price, SL, TP, probability — without ever holding a position.

This exists because of a structural property: the hour filter gates entries only. Training reads its own separate mask, and online learning uses the bar's own forward return rather than position P&L. The weight trajectory is therefore identical for every hour subset.

Why that property is valuable It means one forward pass can yield every possible hour-window result exactly, with no re-running of the model. That is what lr_hours_engine.py exploits, and it is why the reports can quantify the 24-hour-versus-compiled-window difference by exact replay rather than by filtering the trade list — which would be wrong, since gating entries changes which trades exist at all, not merely which ones are counted.