ks-cascade round 1 preregistration: non-inferiority of the routed cascade at the midpoint CPU cost, and a paired AUROC test of the medium's recall state as a routing signal

preregistration · unverified · machine-checked · Kannaka · 2026-11-04T23:58:14.251Z

Preregistered before any arm runs. Full text, data rules, instrument, heads, arms, gate, predictions and the
list of what the adversarial review changed are in spec-prereg-cascade.md (hashed below); this body is the summary.

Data: LongMemEval-S, 500 questions (multi-session 133, temporal 133, knowledge-update 78, single-session-user 70,
single-session-assistant 56, preference 30; 30 abstention variants share a stem with a parent). Split on question
stems with siblings kept together, stratified by type plus abstention: SELECT = the 30 standard questions and their
siblings plus a seeded draw to about 200; CONFIRM = the rest, about 300, never touched by any head, threshold or
human reading until the freeze amendment is on the ledger. The two existing LAYA heads are forbidden (they saw
470 and 78 of the questions); every head is retrained from its base on SELECT rows only.

Instrument: both generators served from a dedicated ollama instance under a cgroup CPU quota, experiment-only tags
with num_ctx 8192, num_thread 6, temperature 0, seed 0; the native API records prompt_eval_count and durations and
a per-call assertion catches truncation. The gated cost is a deterministic token model, prompt tokens times c_p
plus completion tokens times c_g per model, calibrated in an idle window (10 repetitions, interleaved, prefix-cache
samples rejected), validated per row by the cgroup usage_usec delta. Correctness: one judge for every arm, Claude
Sonnet 4.5 through the budgeted gateway key, cached per unique answer; qwen2.5:14b re-judges everything as a $0
sensitivity column. The first answer per (question, model) is canonical and hashed at write.

Heads: F0 = cosine control (top-1, margin, spread, entropy, cosine clusters, crowding, sessions, store size,
haystack and question tokens, route, dates, excerpt length). F1 = F0 plus at most eight HRM features from a frozen
priority list (rank disagreement and Jaccard between the medium's order and the cosine order over a 60-pool, xi
and energy and temporal re-rank factors, medium cluster spread, belief ring order, resonance margin and entropy if
a per-row resonance distinct from similarity exists), all normalised within question, vacuity and leak tests
applied before the freeze with UNTESTABLE as a named outcome. Router label: the 8B's answer judged correct.
Logistic regression, GroupKFold by stem, every SELECT score out-of-fold, the deployed router is the fold mean;
tau* is the largest 8B share whose 80 percent bootstrap lower bound of out-of-fold accuracy loss is at least
minus 0.02. LAYA and TinyJev are fine-tuned on the same rows and replace logistic only if their pooled out-of-fold
AUROC lower bound beats logistic's point estimate; the HRM test uses logistic on both sides regardless.

Arms: A0 always-14B (incumbent), A1 always-8B, A2 oracle (reference), A3 router on F0 (primary), A3-0 regex route
(control), A4 router on F1 (HRM arm), A5 random router at A3's share (control, 10 seeds), A6 and A7 grounding and
abstention arms (exploratory), R1 plain qwen2.5 pair on std30 (reference), R2 the recorded Sonnet std30 row.

Gate, evaluated once on sealed rows by a hashed scoring script, stratified paired bootstrap of 10,000 draws:
G1 (ship): one-sided 95 percent lower bound of accuracy minus A0 at or above minus 0.05, and mean token-model
CPU-seconds at or below half the sum of A0's and A1's means. G2 (router value): routing gain over the matched
random mix with a 95 percent interval excluding zero, and the cascade beats both A1 and the regex route with lower
bounds above zero. The cheapest arm passing G1 ships; a router arm also needs G2; if A1 alone passes, the record
says the 8B suffices and the router claim is void. H1 (the HRM claim): AUROC(F1) minus AUROC(F0) on CONFIRM at or
above plus 0.05, interval excluding zero, above the 95th percentile of 1,000 within-type permutations of the HRM
block, DeLong p below 0.05; between 0 and 0.05 is suggestive, an upper bound below 0.05 is the published null.
G3 (only if H1 passes): A4 beats A3 paired. G4 (grounding, non-shipping): head AUROC at or above 0.75 on
non-abstaining answers and plus 0.05 over the regex-plus-length control.

Predictions on record: A0 scores 0.62 to 0.74; A1 is 0.08 to 0.15 below; c8/c14 is 0.45 to 0.55; logistic
out-of-fold AUROC on F0 is 0.65 to 0.75 and neither text head clears the rule; G1 for A3 is close to a coin flip;
H1 fails with a difference of 0.00 plus or minus 0.03; the smoke check finds no per-row resonance distinct from
similarity at shipped defaults.

Pre-run facts (prerun.json): dataset sha256 08d8dad4...; split.py 8a18f273..., seed "ks-cascade", SELECT 200
(sha256 32bcb939...), CONFIRM 300 (sha256 cde9846a...), 29 sibling pairs, the 30 standard questions all in
SELECT; abstention gold check 20 of 30 matched the regex, fixed by kannaka-bench PR #25, commit 7cf644da...;
kannaka 0.16.17 binary sha256 782d114f...; experiment tags ks-cascade-14b (Modelfile 62511d38..., ollama
ebf1d622ac50) and ks-cascade-8b (Modelfile a4dab95c..., ollama ed092dfd1914), num_ctx 8192, num_thread 6, seed
0, temperature 0. The feature schema, smoke check, cost constants and pilot rates are amendment 1, filed before
any feature matrix is built.

Cost: generation 1,060 calls of niced CPU over about 8 to 14 night windows, $0; heads about $1 to $4 of GPU
(cap $6.21, the balance; $10 needs the maintainer's yes); judge about $2 to $3 on the budgeted key. Sixteen
calendar days. Nothing has run. The maintainer's yes to run is separate from this record.

Written by Claude Code in a session run by the maintainer.

Evidence

content hash e79bd2102e2b4747ef5bcef309f8f382f4ddaf662726586032dd21607e6f89e0

Machine check: machine-checked

Derived from verify_record.py at b3ba75d; it never changes the standing.