How the App Works

Module layout

The package is split so that everything interesting is testable without a display. Only app.py and pages/ touch Tk.

The ten screens, in sidebar order, which is also pipeline order:

Why every stage is a subprocess

This is the central architectural decision and it is not an accident of convenience. The pipeline scripts are imported nowhere. They are always executed as child processes. Three reasons, in order of importance:

The task/manager/chain model

The window never blocks

JobManager runs the child on a background thread that reads the pipe line by line and pushes each line into a queue. The Tk event loop polls that queue on a timer (every 80 ms) and appends whatever arrived to the log console. The UI thread never waits on the child, so the window stays responsive and repaints smoothly during a search that runs for half an hour.

The line history is a bounded ring buffer of 6,000 lines, so a search that produces hundreds of thousands of output lines cannot grow memory without limit.

Why the completion callback needs marshalling

The same applies to threading.Timer, which fires on a thread of its own. call_later() therefore prefers the marshalled path so that both the wait and the callback stay on the GUI thread.

The chain restores its handler

Chain works by swapping in its own advance callback as the manager's completion handler. The handler that was there before — the app's own _on_job_done, which clears the busy state and repaints — must be put back when the chain finishes.

Forgetting that produces a distinctive bug: after running the full pipeline, every subsequent single job would finish without ever telling the UI, leaving the window stuck showing "running" forever. The chain keeps the previous handler and restores it, and a _finished guard prevents a double completion when a cancel and an advance race.

Stopping on failure is deliberate

The chain stops at the first non-zero exit. Every later stage reads an earlier stage's artefacts, so continuing after a failed search would silently build a portfolio out of yesterday's configs — producing a plausible-looking result that has nothing to do with the data you just validated.

Cancellation

Cancelling has to kill the child and everything it spawned. Both script pools mean the immediate child is only the parent of a tree.

Killing only the immediate child would leave every pool worker alive, holding every core and keeping the output pipe open — so the log would never end and the app would never believe the job had stopped.

Text encoding: why the pipe is pinned to UTF-8

The pipeline prints ·, — and ->. Python's console codec on Windows is cp1252, not UTF-8, so both sides of the pipe are pinned explicitly:

This was a real defect before it was a policy. With text=True and no explicit encoding, the parent decoded using its own codec, so under a cp1252 launch the log showed replacement characters instead of the actual glyphs — and in the worst case raised inside the reader thread, which stalled the log silently. The parent also takes precedence over whatever the launching shell had set. Both variables are needed: PYTHONUTF8 switches the default codec, PYTHONIOENCODING pins the stdio streams.

The child also gets PYTHONUNBUFFERED=1, so progress lines arrive as they happen rather than in 4 KB blocks.

Freshness, not just presence

The most dangerous failure in this workflow is not a missing file. It is "the file exists but was produced from different data". The repository holds identically named EAs and tail folders from many earlier runs, sitting side by side, and an old .ex5 will happily sit next to a new .mq5 and look perfectly compiled.

So state.py derives every stage's status from artefact mtimes and flags stale when an input is newer than an output.

The Dashboard colours stale stages amber, the EA page refuses to present a stale .ex5 as compiled, and --check prints the same list.

Progress without guessing

The Optimize page does not show a fake spinner. It parses the search's own structured output and reports a real fraction.

Log lines are also colour-classified from their content:

How the app chooses an interpreter

Settings.python_exe() resolves in a deliberate order:

The runner is located by searching several plausible roots — the PyInstaller unpack directory, the directory of the frozen executable, the package directory, and one level above it — because both "one-file exe" and "drop the exe in the repo" are equally valid shipping layouts.

Output root and naming conventions

These come from the pipeline, and the GUI mirrors them exactly rather than inventing its own.

MetaEditor's two traps

Both are handled in mt5.py, and both are worth knowing because they produce badly misleading failures if you compile by hand.

The compile log is shown verbatim on the EA page, with real diagnostics also extracted into a table of severity / code / file / line / message.

Broker-specific handling

Two things vary per broker, and both are handled explicitly rather than assumed.

The symbol suffix

Brokers decorate instrument names — XAUUSD may be XAUUSDb, XAUUSD.a, GOLD. The pipeline works in the clean name for all filenames and output folders, and the suffix is applied only where the terminal is involved.

The downloader is the one place this must not be confused, because the API only answers to the broker's spelling while the output file must carry the clean name. It has a specific quirk: it names each file after the raw --symbol it was handed. So requesting XAUUSDb writes XAUUSDb_M5.csv — a name no later stage looks for. The GUI passes a folder, then renames the result to the clean name once it lands.

Multiple terminals on one machine

Two installs of MetaTrader 5 keep entirely separate %APPDATA% data directories. The app pairs each install with its data directory by reading that directory's origin.txt, which records the install it belongs to. Orphaned data directories — a leftover pointing at a path that no longer exists — are filtered out.

The chosen terminal's data directory is then passed to every job as MT5_DATA_DIR and MT5_TERMINAL, so the downloader and exporter talk to the terminal you selected rather than whichever one happens to be running.

Threading and lifecycle summary