Sentinel Suite · NinjaTrader 8 · living document·Docs home →

Sentinel Dataset Dictionary

The authoritative reference for the syntax and nomenclature of every Sentinel dataset — the training corpus the Council learns from. If you are about to read, write, ingest, or fit against a .jsonl under Sentinel\Excursions\, this is the map.

This doc is generated from live source (BarsTypes\SentinelFlux_v1_0_0.cs, AddOns\SentinelCore.SystemBuilder.cs VoterCatalog, Indicators\Council_v1_0_0.cs, and Sentinel\Lab\). When those change, regenerate. Related: ML spec · Flux bars spec · Process Atlas.

---

1 · The naming grammar

Everything keys off a scope. A scope is one chart's worth of decision context — exactly the coordinate a model is defined over.

GC  .  212201v12x48  @  STB24FCA
─┬─    ─────┬──────     ───┬────
inst      bartag          lane
└──────── scope ──────────┘        scope = inst . bartag [ @lane ]

Why bartag and not "5-minute": bar granularity determines label optimism. A Renko-labelled row and a minute-labelled row do not describe the same world, so the Lab never pools bartypes — the bartag is a hard partition key, not a cosmetic tag.

Two IDs stitch a single fire's records together across files:

---

2 · Bar-type ID registry

The 212200–212299 block is reserved for the Sentinel bar-type family (declared in every Sentinel bars header). Those three are the ones that build a real corpus:

Other IDs seen in older corpus files (present for completeness — these are not the Sentinel family and several are frozen/legacy):

---

3 · The dataset families

All live under Sentinel\Excursions\. One writer, one schema per folder — the Council recorder (SentinelExcursionRecorder_v2_0_0) owns the council\ tree; the Candidate recorder (SentinelCandidateRecorder_v1_0_0) owns the candidates\ tree; legacy per-sensor recorders default OFF.

⚠ Never pool council\ and candidates\. They answer different questions: the Council corpus is a fusion-gated sample ("the Council chose to fire"); the candidate corpus is the unfiltered clock-native population ("every brick close") — the only one that can measure whether the CLOCK itself carries a base rate. The signal field (COUNCIL vs CONT) and source (council vs candidate in the DB) keep them apart.

Filename anatomy (corpus): 20260715T090945__GC__212201v12x48@STB24FCA.jsonl

• 20260715T090945 — when the recorder instance loaded (session), yyyyMMddThhmmss. Not per-row.
• __ — double-underscore delimiter.
• 212201v12x48@STB24FCA — the bartag, carrying @lane when the chart is laned.
• .jsonl — one JSON object per line.

---

4 · Schema-1.3 corpus row

One row = one Council fire and everything that happened to it. Fields group into four blocks.

4a · Identity / keying

4b · The decision — what the Council said at fire

4c · Context modulators at fire

4d · The OUTCOME / label — what happened after (all excursions in ticks)

---

5 · Tick sidecar (ctick.2)

A two-part file, one per fire. Line 1 = header (join keys + tick-resolution outcome), lines 2…N = the path. (ctick.2 = ctick.1 + the recVer/coreVer/barLabel provenance block, matching the row schema-1.4 bump; ctick.1 sidecars stay valid and load unchanged.)

Header fields of note: recVer/coreVer/barLabel (provenance, ctick.2) · fireId · episodeId · scope · firePx · conviction/sizeMult · barrierTicks · maxFavTicks/maxAdvTicks · msToMaxFav/msToMaxAdv · msToTargetR/msToStopR · firstTouchTick / ftAmbigTick · ticks (count) · trunc (buffer overflowed).

Path rows: {"ms": <ms since fireTime>, "px": <last-trade price>}.

The row's firstTouch (bar resolution) and the sidecar's firstTouchTick (tick resolution) are the same label at two fidelities. The Lab prefers firstTouch; the tick path is for grading fill quality and msToFirstTouch.

---

5b · The CLOCK-EDGE candidate corpus (cand.1)

The second oven (2026-07-17; hypothesis in SENTINEL_CLOCK_EDGE_HYPOTHESIS.md). Tests whether the sampling clock, not the fused voters, carries the edge. SentinelCandidateRecorder_v1_0_0 fires one candidate per brick close — direction = the brick's own direction (the simplest CONTINUATION primitive), no gate, no fusion. Same fidelity spine as the Council recorder (realtime-gated, provenance-stamped, tick sidecar, window-streamed). Derive rule × regime × exit labels late; de-overlap in the Lab (adjacent candidates autocorrelate — use effective-N, not raw count).

Row = schema cand.1, kind:"candidate", signal:"CONT". Shares the Council row's IDENTITY block (recVer/coreVer/barLabel/inst/bartype/fireTime/firePx) and the entire OUTCOME/label block (§4d: maxMFE/maxMAE, milestones, barrierTicks, barsToTargetR/StopR, firstTouch, endReason). It has no vote vector / conviction / episodeId (there is no fusion). It ADDS:

All context is consulted, never fused — each field is a Lab-side retroactive filter, never a hardcoded gate.

Sidecar = candidates\ticks\<fireId>.jsonl, kind:"candidate_tickpath" (schema ctick.2) — same shape as §5 but carries runLength and no conviction/episodeId. Row↔sidecar join on inst+bartype+fireTime (exactly one brick closes per bar time). See §9b for the DB columns; the Lab reads source='candidate'.

---

6 · Decoding the reasons audit string

Built in Council_v1_0_0.BuildReasons(). Example:

EYE~ TRND▼ CCI▼ ADX▼ ENV▼ BRK▲ CMP~ IMKT▲ WAE~ GREV~ STF▼* · clk:Midday · vol×1.3 · roster 10/10 ?STF · denom 4.1/7.8

---

7 · The voter catalog

The source of truth is SentinelCore.SystemBuilder.VoterCatalog, which the Council emits on load to the shared file Sentinel\Models\catalog.conf (tag|role|kind|defWeight|display|seam, one line each). The Python Lab reads that file, so both sides build the fit's feature columns from the same map — no more hardcoded voter list drifting behind the Council. Weights below are catalog DEFAULTS — a scope's Roster.conf (and the Council's F6) override them (e.g. live GC runs CCI at 1.2, not 0.8), and the Lab's fit baseline uses the recorded voteW{} per scope, not these defaults. KIND sets how a voter counts toward the conviction denominator: STATE always dilutes; TRIGGER dilutes only when it fired or is absent (a quiet trigger is absence of evidence, not evidence against).

7a · Weighted voters (the Roster.conf surface)

7b · Context axes (consulted, NOT in Roster.conf — toggled via the Council's Consult* settings)

---

8 · How train.py consumes it

Sentinel\Lab\train.py — the offline fitter. Nothing here touches bin\Custom; it reads the corpus and emits Sentinel\Model.candidate.conf (gated to Model.conf only with --promote).

python train.py --inst GC --bartype 212201v6x24 --barrier 20 --cost 1.5

Pipeline (dataset.py → labels.py → train.py):

1. Load — load_jsonl(council\<schema>\) reads every row where kind=="excursion", then filters by inst + bartype. Never pools bartypes (§1). council_rows keeps signal=="COUNCIL".
2. Label — make_labels: prefers firstTouch (+1→WIN, -1→LOSS, 0→CENSORED); for old 1.2 rows falls back to the pessimistic mfeN/maeN milestone. Censored rows are dropped from the fit.
3. Uniqueness weight — uniqueness_weights (AFML ch. 4): two fires 90 s apart share most of their forward window and are not independent. effective_n = Σ(avg uniqueness) is the N you may quote.
4. Fold by direction — the model feature is x_i = voteᵢ × dir: "did the voter agree with the taken side." Halves the feature space and makes a fitted coefficient directly comparable to WeightEye — it drops straight into Model.conf.
5. Phase 1 — calibration (needs only conviction, so schema 1.2 is enough): Platt-scale conviction → P(win) under purged walk-forward CV; expectancy_floor = the lowest conviction whose calibrated win-prob clears breakeven after cost → the learned ConvictionFloor (replaces the hand-set 0.20). nan = "no conviction level pays" — a real answer.
6. Phase 2 — weights (needs ≥150 rows carrying the votes{} vector): the voter set is data-driven per fit — active_voter_tags admits the catalog voters this bartype actually records above a support threshold (max(30, 5% of vector rows)) with ≥1 non-neutral vote, in catalog order. Rosters range 2→18 voters across bartypes, so a fixed list would drop real voters (e.g. STF) or fit all-zero columns; under-supported and undeclared tags are reported, not silently dropped. Then ridge-logistic on the folded features + orthogonal axes (clockPhase, log_rvol, mtfBias×dir, levelInPath, activeW, voters); grid over C; OOF AUC vs the recorded-voteW baseline. Fitted w.TAG values rescaled to the current weight total (conviction is scale-invariant) → the learned weights.
7. Emit + gate — writes status=ok|reject (reject if no profitable floor, or the fit didn't beat the hand weights). --promote refuses to overwrite the live Model.conf on reject.

Which fields feed which stage

---

9 · Quick rules

• Never round-trip a .jsonl through PowerShell Get-Content/Set-Content — it double-encodes UTF-8 (▼→â–¼). The files are clean UTF-8; the mangling is a display artifact of the cp1252 console.
• Never pool bartypes or schemas in a fit — different bar construction = different label optimism.
• Never pool coreVers (schema ≥ 1.4) — a different SentinelCore/recorder version = potentially different fusion logic. A null coreVer = pre-provenance 1.3; treat it as its own (older, unknown-logic) cohort.
• One writer per corpus folder. The Council recorder owns council\; legacy recorders stay OFF.
• A .jsonl row is only trustworthy when written in State.Realtime — replayed (historical) verdicts carry lookahead. ⚠ This is NOT self-enforcing: the schema has no realtime flag, so a replay leak (a fireTime far in the past) sits in the corpus looking identical to a live row. Verify with Lab\health\corpus_probe.py (stale_dated_rows), and window-scope the fit by fireTime — do not assume the folder is clean.

9a · Proving the corpus is trustworthy — corpus_probe.py

Lab\health\corpus_probe.py (read-only, mirrors probe.py) is the "prove the recording" surface — it never opens the corpus at record time, so nothing else validates it. Run python corpus_probe.py (or --loop 300). It writes corpus_integrity / corpus_folder / corpus_events to sentinel.db (Grafana-readable) with: reconciliation (Ledger fires ↔ excursion rows ↔ tick sidecars, joined on inst+fireTime+firePx+dir; a recon_gap, not pass/fail) · schema hygiene (mixed-schema / unexpected-for-folder, rows_13 vs rows_14) · provenance coverage (prov_coverage_pct + distinct coreVers — a silent logic change shows as two versions in the pool) · silent-loss (trunc sidecars, malformed lines) · stale_dated_rows (the realtime-leak detector). First live run (2026-07-16) flagged ~7,671 stale-dated rows (~83% of the 1.3 corpus, oldest fireTime 2025-12-09) — a replay/backfill leak that had been invisible.

9b · The DB warehouse (sentinel.db) columns

ingest.py reads council\1.3\ + council\1.4\, candidates\cand.1\, and the ticks/council\ticks\/ candidates\ticks\ sidecars. The trades table carries a source discriminator (council | candidate | manual), provenance (rec_ver, core_ver, bar_label, scope) + the context that used to be dropped (regime, adx, conv_bucket, agree, disagree, voters, end_reason, milestone curves + barsTo* in one milestones_json blob). Candidate columns (source='candidate'; NULL for council/manual): run_length, rvol, vol_z, climax, dry_up, clock_phase, mins_to_close, mtf_bias, flux_dir, flux_pressure, flux_diverg. Candidate row↔sidecar reconcile on inst+bartype+entry_utc (mirrors the council row folder; no vote vector). Indexed on (inst,bartype), episode_id, src, schema, bartype, entry_utc. The ingester prints a drop tally by reason (silent loss made visible). train.py reads the JSONL directly, not the DB — the DB is the derived warehouse (see SENTINEL_DB_MIGRATION_SPEC).