Model Ops

The scorecard is not a page, it is the operating system for model trust. Four standing systems decide whether a valuation change ships, whether the world has shifted under the model, whether production is honest right now, and whether known-hard addresses still behave. All are runnable (pipeline/), CI-wired, and write machine-readable artifacts to pipeline/out/.

Promotion rule — champion / challenger (Phase 4.2.1)

No valuation change ships on vibes. Every candidate config runs in shadow mode against the champion over a fixed 100-subject set with cached comp pools (tests/golden/shadow_subjects.json, offline + deterministic), stratified across South Side SFH, North Side 2–4s, suburban Cook, and the $1M+ tail.

A challenger is promoted if and only if its MdAPE is no worse than the champion’s by more than 0.5 points on every segment. A single-segment regression blocks the promotion.

pipeline/shadow_eval.py emits pipeline/out/shadow_report.json and exits 0 = promotable / 1 = blocked (CI-gateable). The engine entry is comp_engine.arv_waterfall; a challenger is injected by overriding the versioned constants for the duration of its run. Champion = registry constants.v2-2026.07. The report states its scope explicitly: this harness validates offline comp-engine ARV constants only. It does not validate score weights, Fair-Housing proxy removal, API contracts, or subject-identity serving policy; those have separate gates.

Release registry, promotion, and rollback

The constants registry above governs hand-set numbers. The separate model release registry governs the complete serving release. Its source manifest is packages/model-registry/registry.json; GET /api/model-registry publishes the current champion ID, exact aggregate artifact SHA-256, gate counts, release scope, and rollback pointer.

Every release row contains immutable per-artifact hashes, provenance, evaluation gates, promotion status, promoter, timestamp, waiver (if any), and rollback metadata. CI recomputes every hash, verifies the aggregate digest, requires exactly one champion, and confirms its model/serving versions match worker.js. scripts/update_model_registry.mjs refreshes hashes after an artifact regeneration but cannot promote or change a model version.

The private GET/POST /api/admin/model-registry operator rail registers candidates and atomically updates the champion alias. A required non-passing gate blocks promotion unless the operator provides a specific waiver of at least 40 characters; the waiver and reason are preserved in the audit table. Rollback can target only a prior retired/rolled-back release and also requires a reason. The current observational-beta champion records, rather than hides, its fairness error-level failure and pending realized-rent validation; the public serving mitigations and disclosure are part of the recorded waiver.

Drift tripwires (Phase 4.2.2)

pipeline/drift_monitor.py → pipeline/out/drift_report.json (status: ok | warn | alert).

Signal	Source	Warn	Alert
Input — per-township Cook sale-price median, last 90d vs trailing-12mo	Cook Assessor sales	median shift >7%	>15%
Output — ARV/ask band by segment	live `/api/listings`	band shift >10pts vs baseline	—

Input drift also reports p25/p50/p75 quantile deltas and volume by area. The output band persists a baseline on first run and compares thereafter. Latest live run flagged ALERT: county median sale price +8.6% recent-vs-baseline, several townships >15% — partly spring seasonality, partly genuine appreciation; the tripwire fires rather than silently absorbing the shift.

Synthetic canaries (Phase 4.3.6)

pipeline/canary.py fires underwrite calls at the live production API across 20 dispersed tracts, both counties (North/West/South Chicago + suburban Cook + six Lake), asserting per probe:

Latency budget (report-only): p50/p95 vs a 3000ms budget, breach count.
Non-null-or-honest-error: a value, an honest suppression with a reason, or a named error — via a machine-readable taxonomy (ok_valued, ok_suppressed, err_*, http_*, rate_limited, timeout, malformed).
INV-5 compliance, version-aware: the structured {method, confidence, as_of} triple when the Phase-2 contract is deployed, tolerantly accepting the current Phase-1 shape with a note (auto-tightens on deploy).

Alerts route to Telegram when TELEGRAM_* secrets are present, else stdout. Writes pipeline/out/canary_report.json. Latest live run: 20 probes, p50 1168ms / p95 1872ms, 0 budget breaches, 0 hard failures, 0 INV-5 breaches; status WARN only because the public API rate-limited 5 of 20 probes (a soft signal, not an outage).

Golden reference set (INV-8, Phase 4.2.3)

tests/golden/golden_set.json locks the audit’s hard addresses to expected-behavior envelopes — never exact prices — run through the real worker engine offline via recorded Socrata fixtures (tests/golden/fixtures/, frozen clock → identical URLs forever). Refresh with node tests/golden_test.mjs --refresh-fixtures.

Subject	Locked behavior
7351 S Prairie Ave	deep-value SFH values (Tier 1/2), ARV > as-is (no 1.22-style undershoot)
1841 W Superior St	hot West Town values, no spurious suppression
7950 S Parnell Ave	Englewood thin — suppression OK; if valued, confidence ≤60
1632 W 100th Pl	Beverly-adjacent — values or suppresses, never blends
1638 N Campbell Ave	golden #1 no-blend: emits `value_as_is`; dispersed renovated evidence suppresses property ARV and keeps neighborhood premium in area context only
1846 N Orchard (luxury)	$5M+ → Tier-1 comps only or suppress, never Tier-2 premium
1834 Sheridan Rd (Lake)	ARV suppressed; tax branch = `lake_median_rate`

Current status: 50/50 plus the cross-subject anti-collision check, offline and deterministic.

Standing invariants — valuation, scoring, Fair Housing, and identity

Executable in tests/invariants.mjs (hermetic: recorded fixtures + frozen clock), delegating INV-8 to the golden test. A red invariant is shown red.

#	Invariant	Enforcement point
INV-1	Comps pool follows the declared expansion ladder and never exceeds 1.0mi; subject tract resolved	`tests/invariants.mjs` (worker Tier-1 pool + comps metadata); worker runtime log
INV-2	Cross-surface equality: ARV, market rent, flip_net identical on underwrite vs property	`tests/invariants.mjs` (N fixture subjects); nightly sample
INV-3	No defaulted unit counts	static grep of `worker.js` + runtime `units_required` probe
INV-4	No top-decile score with failing gate economics	`tests/invariants.mjs` — LIVE: `underwrite.strategy_scores.*` gated on their own economics are capped at `gate_ceiling` (40); a flip that loses money at ask / DSCR<1.0 can never reach the top decile
INV-5	No displayed valuation without {method, confidence, as_of}; Tier-3 renders no number	`tests/invariants.mjs` (underwrite/property/listings) + live `canary.py`
INV-6	No serving table updated by a batch that failed its expectations (quarantine only)	`pipeline/expectations.py` poisoned-batch check
INV-7	Every pricing constant resolves to the versioned registry (no literals)	`tests/invariants.mjs` parses `worker.js` VAL_CONST vs `constants-registry.json` (21 constants + registry_version)
INV-8	Golden addresses behave within locked envelopes	`tests/golden_test.mjs` (delegated)
INV-FH	Restricted income/crime proxies are score/risk-inert and absent from public tract payloads	`tests/invariants.mjs`; this does not replace the separate fairness regression, which currently fails its error-level gate
INV-ID	Canonical subject facts never borrow nearest-comp GLA/year/beds; artifact subject parity and hash perturbation hold	`tests/invariants.mjs`

CI wiring

ci.yml (push/PR): deterministic gates golden_test + invariants, model-registry artifact/version integrity, analytics privacy-boundary tests, worker/ground-truth/smoke, Python py_compile sweep + offline comp_engine fixture run + module dry-runs; apps build as a separate job.
nightly-validation.yml (cron 09:30 UTC): canary + drift + incremental flip-grading + monthly scorecard guard; artifacts uploaded; Telegram on failure only when secrets are present. Neither workflow deploys anything.