Step 1 of 7

Data

Everything in this system is derived from one CSV. A wrong header, a descending sort, or a zero-filled spread column does not crash anything — it silently optimises the wrong thing. This step exists to make that impossible.

What this stage does

Two things, and only one of them is optional: get bars into data/<SYMBOL>_<TF>.csv, and validate them against the assumptions the engine relies on.

Broker terminal

→

CSV

→

Validator

→

data/<SYM>_<TF>.csv

The file contract

Exactly this header, in exactly this order:

time,open,high,low,close,tick_volume,spread,real_volume

Timezone

Broker server time, naive (no offset, no Z)

Format

YYYY-MM-DD HH:MM:SS

Order

Oldest → newest. Strictly increasing.

Path

data/<SYMBOL>_<TF>.csv

Symbol / TF source

Parsed from the filename — data/XAUUSD_M10.csv means symbol XAUUSD, timeframe M10

⚠

Server time, not UTC This is the single most consequential detail on the page. Every "best hours" window, every hour feature, and every session filter is expressed in the broker's server time. If the CSV is in UTC, the hour features are shifted and every hour window is wrong — with no error anywhere.

The API downloader returns UTC and therefore has an --offset flag (default 3.0 for a UTC+3 broker). The terminal-side exporter reads rates[].time, which is already server time, so its shift defaults to 0 and should normally stay there.

Three ways to get the data

All three are in real use. They are not equivalent.

Export from the terminal — recommended

scripts/ExportOHLC_MQL5.mq5 runs inside MetaTrader. The app installs and compiles it for you, then shows what landed in MQL5\Files.

Why it is the best option:

It uses this broker's own history, not a mirrored feed.
It uses this broker's own per-bar spread — which is what every cost figure downstream is computed from.
rates[].time is already server time, so there is no UTC→server offset guess to get wrong.
No wine python, no MetaTrader5 pip module, no DLL.

Output goes to the terminal's MQL5\Files folder (or Common\Files). Import it into the repo, validate it, done.

ℹ

The script writes a sidecar <file>.csv.summary.txt next to the CSV with the bar count, first/last time, median spread, count of zero-spread bars, session gaps and a ready-made "next steps" block. It flags SHORT-HISTORY when the broker simply does not have as many bars as it was asked for.

Key inputs: InpBars (default 10,000, minimum 100), InpExtraTimeframes (e.g. "M5,M10,M15,H1"), InpSpreadFallback for bars reporting zero spread, and InpSymbolLabel — which should be set to the clean name when the broker's symbol is decorated, so the file is named XAUUSD_M5.csv rather than XAUUSDb_M5.csv.

Download via the MetaTrader5 Python API

download_ohlc_all.py fetches the last N bars through the official Python module and writes pipeline-format CSVs.

download_ohlc_all.py --symbol XAUUSD --tf M5 --bars 10000 --offset 3.0 --out data

Requirements: a Windows Python with the MetaTrader5 package — the module talks to the terminal through a Windows DLL. On Linux that means the python inside the wine prefix. The terminal must be running and logged in.

⚠

The --symbol naming trap The script names each output file after the raw --symbol it was given. Requesting the broker's XAUUSDb writes XAUUSDb_M5.csv — a name no later stage looks for. Every stage reads the clean XAUUSD_M5.csv.

Left alone, this would make the next stage read a stale XAUUSD_M5.csv from some earlier run, silently. The GUI therefore passes a folder for --out, then renames the result back to the clean name once it lands — and emits a loud warning if the file is not where it expected.

Import an existing CSV

You exported or downloaded it elsewhere. Drop it into data/ with the right name and validate it. Nothing about the pipeline cares where the bytes came from — only that the contract holds.

The validator

check_ohlc_csv.py data/XAUUSD_M5.csv

Exit code 0 means no errors (warnings are allowed). 1 means at least one error was recorded. The validator prints its findings verbatim, because they encode the contract the engine relies on.

What it checks, and why each one matters

#	Check	Verdict and reasoning
1	Header	Exact text and order, byte-compared against the first line. A case-only difference is a warning. Also notes LF vs CRLF.
2	Row count / size	error if empty or under 500 rows.
3	Time	Must parse, be strictly increasing, contain no duplicates and no backwards steps, and align to the timeframe grid — `seconds % (step_min × 60) == 0`.
4	Bar size vs the filename's TF	Computes the modal inter-bar step in minutes and errors if it does not match the timeframe in the filename. This is subtler than it looks. A grid-alignment test alone would pass, because every M10 timestamp is also a valid M5 timestamp. So the modal step is used instead, and a 2× mismatch gets a specific message: the file very likely holds the double timeframe.
5	OHLC sanity	All finite and positive; `high ≥ max(open, close)`; `low ≤ min(open, close)`, with a `1e-9` tolerance.
6	Gaps	Counts steps longer than one bar and reports the largest hole plus the first three. Gaps of 30 hours or more are identified as weekends or holidays rather than flagged. Prints the total span in days.
7	Spread	Median, minimum, maximum and count of non-positive values. error if the median is zero — because the search derives `max_spread` from that median and every report would then charge $0 cost per trade. error if every bar reports zero. warning if more than 25% report zero, or if the maximum exceeds 20× the median.
8	Trainability	warning if there are fewer rows than `MAX_TRAINING_BARS + MAX_LOOKAHEAD + FEATURE_LOOKBACK` = 1500 + 8 + 51 = 1559. Below that, the longest allowed training window cannot be filled.
9	Hour histogram	Notes the top three server-time hours. A smell test for a shifted clock — if the busiest hours look wrong for the instrument, suspect the timezone.

Sample output

========================================================================
data/XAUUSD_M5.csv   (XAUUSD)
========================================================================
  ok    header ok: time,open,high,low,close,tick_volume,spread,real_volume
  ok    10000 rows, 812.3 KB
  WARN  3 sessions gaps over 30h (weekend/holiday)  largest 49h
  ERROR median spread is 0 - cost model would charge $0 per trade
  -> no problems found
  -> usable (2 warning(s))
  -> DO NOT OPTIMIZE ON THIS FILE (1 error(s))

1 error(s) total - fix these before training.

The three verdicts

Clean

no problems found

Proceed to the search.

Usable

N warning(s)

Proceed, but read each warning. Gaps and a mild spread anomaly are usually fine.

Condemned

N error(s)

Do not optimise on this file. Fix the data or pick another source.

Why the spread column decides everything

The engine does not charge spread on individual fills. Instead the search reads the median of the spread column and stores it in the summary as max_spread. Every report then estimates cost as:

cost = median_spread × number_of_trades × POINT_DOLLARS

where POINT_DOLLARS = 0.01   ($ per point, per 0.01 lot)

Two consequences follow. First, a zero-filled spread column makes the whole run look profitable by exactly the cost it never charged. Second, the spread you supply is the spread you are backtesting — a broker's indicative spread on a quiet demo server is not the spread you will pay at 08:30 during a news release.

⚠

Check the median spread tile before starting a search The Data page shows it as a KPI, and the validator prints it. For XAUUSD on a typical retail account it is a small number of points; if it reads 0, stop. Reports always show raw and net-of-spread figures side by side so the cost assumption is never hidden.

The Data page

The page mirrors the three routes and adds the checks you would otherwise do by hand.

Current data file

Six KPIs read straight off the CSV: rows, span, first bar, last bar, median spread, and dollars per trade at 0.01 lot.

Route 1 — Export from terminal

Installs the MQL5 exporter into the terminal, compiles it, and lists what appeared in MQL5\Files so you can import the newest one.

Route 2 — Download

Bars, offset and timeframes; shows the exact command before running it.

Route 3 — Import a CSV

File picker straight into data/.

Validate

Runs the repo's own check_ohlc_csv.py and shows its output verbatim, with the verdict label.

ℹ

The page tells you which file the pipeline will actually read Not just "the newest CSV in data/" — the resolved data/<SYMBOL>_<TF>.csv for the symbol and timeframe currently selected in the sidebar footer. If you have data for three instruments and two timeframes, this is the line that removes the ambiguity.

Checkpoint

✓

Before moving to Step 2

The validator reports no problems found, or you have read and accepted every warning.
The median spread is not zero.
The row count is at least 1,559 for the search to use its longest training window.
The bar cadence matches the filename — the modal step equals the labelled timeframe.
The first and last bar timestamps look plausible in server time.

If something is wrong

Symptom	Cause	Fix
"File holds the DOUBLE timeframe"	The bars are M10 but the file is named `_M5.csv` (or similar).	Rename the file to match its true timeframe, or re-export at the intended one.
Median spread is 0	The export route did not populate the spread column, or the broker genuinely reports none.	Re-export with the terminal script (which reads real per-bar spread) and set `InpSpreadFallback` if some bars are zero.
Time is not strictly increasing	The file was sorted descending, or concatenated from two sources.	Re-export. The pipeline assumes oldest→newest everywhere.
Hours look shifted	The CSV is in UTC rather than server time.	Re-export from the terminal (already server time), or adjust `--offset` on the API downloader.
Under 1,559 rows	Not enough history for the longest training window.	Request more bars — the default is 10,000, which for M5 is roughly 34 trading days.

Full troubleshooting catalogue →