Sentinel Suite · Visual guide 2 of 2

Sentinel Runtime Topology

Guide 1 showed what the suite is. This one shows what actually runs, who owns each artifact, and how data crosses a process boundary — because most of the expensive bugs in this project were boundary bugs: a seam orphaned by an F5, a probe reading a rotation generation that did not exist, a second writer on a one-writer file. Companion page: ← Architecture Map

Processes 1 NT + 6 daemons Ports 5 Boxes 2 (main + legacy-node) Persistence files → SQLite Core v1.45.0 · Council v1.11.0

AIn-process — inside NinjaTrader.exe

Everything in this diagram shares one AppDomain and one compiled assembly (NinjaTrader.Custom.dll). That is why one broken .cs blocks the whole suite, and why an F5 that swaps the assembly can leave a chart's bar type publishing into an orphaned static seam store.

NINJATRADER.EXE · ONE APPDOMAIN · ONE NINJATRADER.CUSTOM.DLL · UI THREAD + PER-SERIES DATA THREADS ① PER-CHART PIPELINE (ONE PER SCOPE) ② IN-PROCESS SEAM STORE ③ PROCESS-WIDE ADDONS Bar clock — BarsType TBars · Flux · Drift · Tide Lattice · Effort · TbarsCount Sensors / voters publish a seam · never call up Council v1.11.0 reads seams → one CouncilState Deck | Bridge the only two order sources Recorders row + tick sidecar · realtime-gated verdict SentinelCore seam stores static · in memory only key = <inst>.<barTag> auto-expiring NO history NO as-of semantics Beacon detects an F5-orphaned publisher 25 voter + 5 axis seams write write read SentinelCore.SystemBuilder VoterCatalog · RosterIO · LaneIO — one reader/writer per config file, so the Council (reader) and Cockpit (writer) cannot drift on format Services State · Log · LogEngine · Alert · News · Risk · Arc (fleet) · Binds · Lens (analytics) · Skin (themes, cards, cards.off render gate) Surfaces & tools Cockpit · Dashboard · Copier · Conductor (replay transport) · Quartermaster · NT8BridgeServer (headless compile / deploy) SentinelCore.Foundation Log (6 generations) · Ledger · State · Swallow() — the recorded empty catch size, then validate reads rows + ticks log · ledger · state ↓ ALL PERSISTENCE LEAVES THE PROCESS AS FILES — CONTINUES IN DIAGRAM B ↓
Read it as an inversion: nothing upstream knows a consumer exists. A sensor writes a seam; the Council reads it; the Cockpit reads all of them. That is what makes w=0 auditioning, sensor replacement and standalone grading possible without touching fusion.
The boundary that bites The seam store is static and lives in the AppDomain — not on disk. An F5 recompile leaves a chart's existing bar-type instance bound to the old assembly, still publishing into a store nobody reads. Symptom: a voter that is silently, permanently absent. Detection: Beacon/BeaconForeign, so a consumer can say DECOUPLED — restart NT rather than “absent”. A chart reload does not fix it. Restart NinjaTrader.

BOut-of-process — files, database, daemons, boards

The canon pipeline is tools → JSONL → ingester → DB → analyzer. Everything downstream of NT is plain Python over plain files; there is no service mesh and no message broker, which is deliberate — a bake node that loses power resumes from a checkpoint file.

① ON-DISK ARTIFACTS — ONE WRITER EACH Sentinel\*.conf Roster · Profiles · Lanes · News · Copy · Cockpit · Run Excursions\ council\1.5\*.jsonl (rows) ticks\*.jsonl (sidecars) Ledger\ · State\ BrickLog\ · Snapshots\ Support\ · Conductor\run-log sentinel.log 6 generations (.1 … .6) the live truth — read it CSV tick exports gbNRDtoCSV · BarDump ⚠ stamps are CHICAGO ② PYTHON DAEMONS — EACH SINGLE-INSTANCE-GUARDED ON A PORT Lab\harness\ bar types + fusion, no NT behind the equivalence gate ingest.py --watch JSONL → SQLite the ONLY DB writer for rows health probes probe.py · corpus_probe.py :8502 · heartbeat row/cycle docs\audit.py drift · links · contracts :8505 · every 15 min tail tail + live NT Lab\db\sentinel.db SQLite · WAL · busy_timeout 30 s · single node ⏭ Postgres is the standing debt Grafana · :3000 trades · health · docs · legacy-node OPS — dashboards are generated Streamlit · :8501 Observatory · Explorer · Paths RESEARCH — reads the DB Analysis CLIs pathlab · sensor_truth · voter_edge · train.py reads bypasses the DB ③ legacy-node — REMOTE BAKE WORKER (TAILSCALE) NT + Conductor + recorders autologin → autorun → resume its own corpus + log cards.off — render gate ON replay_sync.py rule-based bake-set sync → compile legacy-node_probe.py · :8504 READ-ONLY over SSH → the same DB
Solid arrow = a write or a tail. Dashed = a read that crosses a machine or skips the DB. Note the two deliberate DB bypasses: the harness works from raw CSV so it can never inherit NT's mistakes, and the analysis CLIs can read sidecars directly — an expedient, explicitly not the architecture. One edge is omitted for legibility: legacy-node_probe.py writes into the same sentinel.db as the local probes — that is the only thing crossing back from the remote box.

CProcess responsibility

One row per thing that is separately startable, killable, or crashable.

ProcessWhereResponsibility
NinjaTrader.exeNTHosts everything NinjaScript: charts, bar clocks, sensors, Council, order sources, AddOn services. One AppDomain, one assembly — so one bad file is a total outage, and an F5 is a code-swap event with consequences.
Chart pipelineNTPer scope: build bars → publish sensor seams → fuse a verdict → optionally place and record an order. The unit of everything; a scope is one chart's worth of context.
AddOn servicesNTProcess-wide, chart-independent: safety gate, governor, kill switch, alerts, news, fleet identity, config IO, theming, copier, replay transport.
NT8BridgeServerNTThird-party AddOn (cli-nt-bridge) exposing NT's own compiler headlessly: real Roslyn errors in ~15 s, plus deploy which recompiles and hot-reloads under whatever is running.
ConductorNTReplay transport — seek, queue, per-session checkpoint, crash-resume, autorun. Holds no account reference and calls no order method, by design.
ingest.py --watchdaemonTails the JSONL corpus into SQLite. The only writer of corpus rows to the DB; folds the council vote-vector in.
probe.py --watchdaemon :8502Samples NT + suite health (incl. swallowed Lab faults) into the DB every cycle. Writes a heartbeat row so 0 means “measured zero just now”, never “dead probe”.
corpus_probe.pydaemonAudits whether the recorded corpus is trustworthy — completeness, schema, vote coverage — as opposed to merely present.
docs\audit.pydaemon :8505Docs drift: broken links, stale HTML siblings, contract-version drift, dangling {{tokens}}, orphans. Gates commits via a pre-commit hook.
legacy-node_probe.pydaemon :8504One read-only PowerShell metrics snapshot per cycle over SSH. Never touches NT on the remote box.
Streamlitserver :8501The research surface — Observatory (honesty ladder), Explorer, Tick-Paths, Council-Paths. Pages import the CLIs so numbers cannot drift between the two.
Grafanaserver :3000Ops boards over the SQLite datasource. Dashboard JSON is generated by build_*_dashboard.py, never hand-edited.
snapshot.pytaskTiered WORM corpus snapshots, validate-before-destruct. Two scheduled tasks. Guarantees integrity — not correctness.
replay_sync.pytoolClassifies the bake set by rule (so new Sentinel files are auto-included), tars, applies, compiles on the remote node.
Harness CLIstoolBar types + sensors + fusion outside NT from raw CSV. Trusted only behind equivalence, which itself is trusted only behind selftest_equivalence.

DArtifact ownership — the one-writer rule

Two loadable recorders once meant a workspace load could silently start a second writer emitting contaminated rows into the training corpus. Ownership is therefore explicit, and “harmlessly present in the tree” is not a category that exists here.

ArtifactWriter (exactly one)ReadersFormat / note
Seam storeseach sensor / bar clock, for its own seamCouncil · Cockpit · Bridge · HelmIn-memory statics. Scope-keyed, auto-expiring, no history.
Excursions\council\ExcursionRecorder v2.3.0ingest.py · pathlab · train.pySchema 1.5. ⚠ 1.3/1.4 are contaminated — never train on them.
Excursions\ticks\the recorderspathlab · tickpaths · limitlabctick.3 — self-describing header, so context needs no join.
Excursions\candidates\CandidateRecorder v1.1.0pathlab --kind candidatesJoins at 97% on (inst,bartype,dir,fireTime).
Ledger\SentinelCore FoundationLens · Dashboard · diagnostics exportOrders/actions/fills + episode & instance context.
sentinel.logSentinelCore.Loghumans · probes · forensics6 generations. Rotation is deliberately silent — logging it would recurse.
Roster.confCockpit ⑤ / System Builder via RosterIOCouncil via the same RosterIOOne reader/writer type ⇒ format cannot drift.
Lanes.confoperatorLaneIO → CouncilWins over the F6 setting, and the override is announced in the log.
sentinel.dbingest.py + the probes (own tables)Grafana · Streamlit · analysis CLIsWAL + 30 s busy timeout, because concurrent writers were real.
catalog.dbquartermaster\catalog.pyQuartermaster CLI · bake planningDates live in the filename ⇒ a 100 GB scan never opens a .nrd.
run-log.jsonlConductorConductor resumeSession-boundary checkpoints. ⚠ a polluted log misdirects resume.
cards.off / theme.txtoperator (file presence)SentinelSkin, polled ≤2 sPresence = render gate off. Seams still publish; only drawing no-ops.

EThreads, lifecycle and ports

RuleWhy it exists
UI thread onlyEnumerating ChartControl.Indicators or touching a window throws off the data thread. NT is multi-UI-threaded — each window has its own dispatcher, so cross-window property access throws every time and needs Win32 HWND calls instead.
Data thread onlyReading ind.Values[i][barsAgo]. Resolve the reference on the UI thread, read values on the data thread.
Never in OnRenderReading Value[]. Cache display values in OnBarUpdate; a render pass must not touch series state.
1 s timer, not the tapeA safety readout that refreshes on ticks displays a stale account in a quiet market. The Deck's account band is timer-driven for exactly this reason.
F5 vs restartF5 recompiles and reloads. It does not reload a chart's sticky bar type, and it orphans bar-type seams. Bars-type work ⇒ full NT restart. deploy hot-reloads under a running instance; a plain file write does not.
Version forksNamespace + class name are an indicator's serialization identity. A new version is a different tool: remove the old instance from the chart first, or two of them collide on one scope.
Realtime gateUpdatedUtc stamps wall-clock even during replay, so a freshness check cannot tell replay from live. Anything that records gates on State.Realtime.
:3000 · :8501 · :8502 · :8504 · :8505Grafana · Streamlit · health probe · legacy-node probe · docs audit. Each daemon binds its port as its own single-instance guard.

FThe six invariants this topology exists to protect

  1. One writer per artifact. A second writer does not corrupt loudly — it emits plausible rows in a slightly different schema, and you find out months later when a model trained on both disagrees with itself.
  2. A crashed sensor must be distinguishable from a quiet one. Fail-open abstention means an indicator that throws in OnStateChange simply never votes, forever, and nothing says so. That is what the declared roster and RosterComplete exist to surface.
  3. Zero is a measurement; absent is not. Every board headline reads a heartbeat table, so an empty findings table can never be mistaken for a healthy zero.
  4. A value that resolves is not a value you can trust. NT reported MaxSpeedValue = int.MaxValue — a resolved-but-implausible number that silently disabled a clamp and reported success. Measure the outcome; do not believe the flag.
  5. Replay must equal live, and that is a measurement. Hence the equivalence gate, its own self-test, the determinism diff, and the realtime recording gate. A harness not yet proven equal to NT is just a faster way to be wrong.
  6. Instrumentation that changes behaviour is not instrumentation. The frozen control strategy is never modified; its instrumented sibling must reproduce its trade list exactly before anything it reports is believed.
Current runtime state Nothing is running and nothing is broken — nt8bridge compile → errors: []. Before any bake or live run: F5 + full NT restart (Core v1.45.0 / Council v1.11.0), remove the old Council from each chart before adding v1.11.0, and re-add Deck v0.2.6. Lattice (212205) and Effort (212206) compile clean but have never been loaded — loading them also needs the restart, because bar types are sticky.