Sentinel Suite · NinjaTrader 8 · living document·Docs home →

Sentinel — Historical Replay & the Fusion Core

How to run the learning loop on historical data so a corpus can bake with the market closed — without poisoning it with lookahead. Companion to the Thesis (the why of the loop) and the ML Spec (the corpus schema + the Lab). Reference chapter of the docs.

1.The problem, and the reframe

The Recorder records Council fires realtime only. With the market closed, the corpus stops growing — and the ConvictionFloor can't be fit until it does. We need to bake in the background.

But "realtime only" is widely misread. Two facts from the code:

The Council only publishes its verdict in State.Realtime and stamps each verdict IsHistorical = (State == State.Historical).
During a normal chart's history load, the Council publishes nothing and the Recorder records nothing.

So the guard is not refusing historical data — it is refusing silent lookahead through the seam, which has no as-of clock (SetCouncilState stamps wall-clock UtcNow even while replaying old bars, so a consumer can't tell a replayed verdict from a live one). Realtime-only ≠ live-market-only. Historical replay is still on the table; we just cannot do it through the bar-history-load path.

2.Three tiers (cheapest first)

Tier 1 — NT Market Replay (today, zero code)

NinjaTrader's Playback connection feeds historical tick data through the realtime pipeline. During the replayed portion State == State.Realtime, so the Council publishes and the full-range Recorder records — tick-accurate first-touch and all. The guard passes because replay genuinely is the realtime code path.

Use it: download a replay day for the instrument, put the Council + Recorder + the sensor stack on a Playback chart, run unattended. Limits: replay-data-gated per instrument/day; runs at replay speed (fine overnight). Validate: confirm COUNCIL rows land with realistic barriers.

Tier 2 — the self-contained Council Replay harness (the feature)

A historical-first Council. One NinjaScript unit that hosts its sensors on its own series, fuses them, and records excursions in a single causal bar pass — no cross-process seam. It runs on plain historical bars (fast, no replay-data dependency) and is deterministic — it removes the cross-instance processing-order race that makes the seam path replay-unsafe. This is §3.

Tier 3 — Databento (the data-quality axis, "Phase 4b")

For honest intrabar fills at scale and deep history. The clean split of responsibility:

NinjaTrader owns the brain. One sensor implementation, in NinjaScript. Never port the sensors to Python — two implementations drift, and the model then learns the drift.
Databento tbbo/trades owns the honest outcome ruler. NT emits fire events (scope · time · price · verdict); an offline pipeline resolves target-vs-stop first-touch on tick truth.

This is the backtest-fill-resolution lesson made infrastructure: bar-level first-touch is optimistic (it took CompressionBase from 81% → 37.5% when it moved to tick fills). Feeds the same schema-1.3 corpus the Lab already reads.

3.The fusion core (the enabling refactor)

The Council's OnBarUpdate does two separable things:

Gather — read each sensor's fresh reading from its seam and derive a directional vote (AddVote(tag, dir, weight, …)), plus the modulator states (Clock, Participation, MTF, Location, squeeze) and the hard-veto flags.
Fuse — pure math: kind-aware denomW → bias (deadband) → conviction = |netScore| / denomW → context damping → sizeMult, plus the agree/disagree tally.

Step 2 is a pure function of its inputs. Extract it:

CouncilFusion.Fuse(
    votes:      list of { tag, dir, weight, kind, counted }
    declared:   the roster (for the kind-aware denominator)
    modulators: { squeeze, clockPhase, inSession, rvol, mtfBias, lvlInPath, voters }
    veto:       { vetoed, reason }              // resolved by the front-end (seam/account reads)
    config:     { biasDeadband, convictionFloor, damp factors, minVoters }
) -> { bias, conviction, sizeMult, agree, disagree, contextMult }

Then there are two front-ends over one core:

Live Council (unchanged behaviour) — gathers votes + modulators from seams, calls Fuse.
Replay harness — gathers the same votes from hosted sensor instances (read each sensor's directional value directly, bar-by-bar), calls the same Fuse.

One fusion truth, exercised historically and run live. (This is also the seam of the generic vote registry — a vote is a vote whether it arrived from a seam or a hosted sensor; see [council-custom-voters].)

4.The correctness gate (non-negotiable)

A historical corpus is trainable only if the verdict computed on bar X equals the verdict that would have been live at bar X. That requires:

Causal sensors — no repaint, no forward-looking state. A sensor that looks ahead poisons every row.
One fusion core — §3, so there is no live-vs-replay math drift.
Deterministic order — hosting sensors in-process fixes the vote/modulator ordering the multi-instance seam path leaves racy.

Validation: run the harness over a window we also have live Recorder rows for, and confirm the verdicts match (same bias, conviction within rounding). If replay ≠ live, the corpus is fiction — do not train on it. Same correctness-precedes-collection discipline as the Thesis.

5.Build sequence

Tier 1 now — bake a first corpus via Market Replay while the rest is built.
Extract CouncilFusion.Fuse (§3) as a pure core; rewire the live Council to call it; F5 + verify live behaviour is byte-for-byte unchanged (careful surgery — market-closed is the right time).
Build the replay harness — hosts the sensor stack, gathers votes from the hosted instances, calls Fuse, records excursions causally on historical bars.
Run the correctness gate (§4) on an overlap window; only then trust replay-baked rows.
Tier 3 / Databento — the honest-fill pipeline for scale and depth.

Status (2026-07-11): specified. Tier 1 usable now. Tier 2 core extraction is the next build.