ks-cascade judge decision: the preregistered fallback (qwen2.5:14b primary, Sonnet 4.5 on the A0/A1 discordant pairs), chosen by the operator before any row is judged

note · unverified · claimed · Kannaka · 2026-11-08T15:14:17.739Z

STATUS
- Decision note for the preregistration 01M6MFZEQBGYD8517QFRP0GHP6 (amendments 1 and 1b). Written 2026-11-08, before
  any answer was judged. SELECT (198 questions, both generators) and the 29-question audit finished generating at
  07:50Z with --no-judge. No verdict, accuracy, or comparison of answers with gold has been computed or looked at on
  any split. The answer files exist in plaintext on the host; they have been checked mechanically (row counts,
  hashes) but not scored.
- Decided by the maintainer at about 14:50Z the same day.

WHAT IS DECIDED
Section 3 names Claude Sonnet 4.5, through the KAX gateway, as the one judge. It also names a fallback "if the key is
unavailable": qwen2.5:14b as the primary judge, with Sonnet re-judging every A0/A1 discordant pair as a co-primary
sensitivity row. We apply that fallback.

DEVIATION, STATED PLAINLY
The trigger has not happened. The key is available: on 2026-11-05 it listed the alias agent-brain, configured to route
to anthropic/claude-sonnet-4-5, and a test call succeeded. This is the operator's choice, not a key failure. The
reason: Anthropic's deprecation schedule retires claude-sonnet-4-5 on the Claude API on 2026-11-30 (source:
https://platform.claude.com/docs/en/about-claude/model-deprecations, read 2026-11-08), and we cannot guarantee that
CONFIRM judging ends before then. Section 3 requires one judge for every arm across SELECT and CONFIRM, so a Sonnet
primary could force a judge change in the middle of the study. A local qwen2.5:14b cannot be retired, and it costs $0.
The fallback's judge roles are used unchanged. This note adds the CONFIRM discordant re-judge, the audit's judge, the
reference row's labelling, the unavailability rules and an exploratory subsample, all written before any verdict exists.
Known directional prior: Prediction 1 implies qwen2.5:14b grades more leniently than Sonnet on this bench. Our b
prediction below says qwen is relatively harsher on the 8B's answers, which would make G1 harder for the arms that use
the 8B, not easier. The switch was chosen for judge continuity; its expected effect on the ship gate is conservative or
unknown, not favourable.

JUDGE ROLES
- Primary, for every arm on SELECT, the audit and CONFIRM: qwen2.5:14b on the dedicated ollama instance on debain2,
  model digest 7cdf5a0187d5c58cc5d369b255592f7841d1c4696d45a8c8a9489440385b22f6. CONFIRM is judged only if the
  instance lists the same digest; otherwise stop. It uses the native /api/chat (think off, temperature 0, seed 0,
  5 output tokens) and the bench judge prompt. Each unique (question, answer sha256) gets one call, and grade() gives
  the final verdict. It runs only while no generation process is alive on that host; if the script's process guard
  matches a non-generating leftover, the override is used only after checking that no bench.answer process exists, on
  the maintainer's confirmation, and the check and the pid are recorded in the run record. Its verdicts feed every
  gate, the router and grounding labels, and the audit (the audit's verdict-flip count in Prediction 11 is a qwen
  count). Ship decisions use qwen verdicts only.
- Co-primary sensitivity row: Sonnet 4.5 through agent-brain, on both answers of every question whose primary A0 and
  A1 verdicts differ. This covers SELECT and CONFIRM, each after its primary judging is complete. It enters no gate.
  "Co-primary" is fixed here, before any row is judged, to mean: (1) published in the same tables and with the same
  prominence as the primary verdicts; (2) G1's accuracy half (one-sided 95% lower bound of dAcc against minus delta,
  same bootstrap) is recomputed for every G1 arm with Sonnet's grade() verdicts substituted on the re-judged pairs.
  Paired dAcc against A0 moves only on A0/A1 discordant questions, so the substitution reaches every row that can move
  it under qwen; concordant pairs keep qwen's verdicts, and that limitation is stated. If the Sonnet-substituted G1
  outcome differs from qwen's for any arm, that arm's G1 result is reported as judge-dependent, with both lower bounds
  side by side. Also published: agreement with the primary on those rows, and how many discordant pairs keep their
  direction. Sonnet's dollars are read from the gateway spend log and reported separately (the summary's list-price
  field reads $0 for a sensitivity judge, and qwen's summary is run at price 0/0); qwen2.5:14b costs $0. Expected
  Sonnet calls: both answers on roughly 15-25% of questions (SELECT about 60-100 calls, CONFIRM about 90-150), plus
  the subsample and the reference row; well inside the $3/day cap.
- Reference row R1 (the std30 row of plain qwen2.5:14b and qwen2.5:7b; not router R1): plain qwen2.5:14b is a
  generator there, so under this fallback it would be graded by itself. R1 is judged by qwen2.5:14b and labelled
  "self-judged (judge = generator)" in every table. Sonnet also judges every R1 std30 answer (at most 60 calls) as a
  labelled second column. R1 enters no gate.
- Disclosure: qwen2.5:14b is A0's base model (kannaka-brain-v3 is Qwen2.5-14B-Instruct plus a LoRA). It is not one of
  the A0/A1 generators, but it could favour its own family. The subsample below measures the arm-dependent part.
- Before each Sonnet session (SELECT discordant, subsample, reference row, CONFIRM discordant), the gateway's upstream
  model for agent-brain is read from the gateway's model info or configuration. The upstream model string, the sha256
  of that configuration line and the UTC time are recorded in the run record. A session whose upstream cannot be
  confirmed as anthropic/claude-sonnet-4-5 is not run, and the row is treated as unavailable from that point.
- Amendment 2's text describes the judges as here (qwen native on the dedicated instance; Sonnet through the gateway)
  and takes Sonnet dollars from the gateway log.

PREDICTIONS AND BOUNDS UNDER THIS DECISION
- Prediction 1 (A0 Sonnet-judged 0.62 to 0.74 on CONFIRM) is not evaluable on CONFIRM as written and is scored "not
  evaluable". The qwen-judged CONFIRM A0 accuracy is reported beside it, labelled "judge changed from the prediction's
  Sonnet", and the subsample's Sonnet-judged A0 accuracy (n = 60, SELECT) is reported as exploratory. Its voice-LoRA
  clause is scored on qwen verdicts, with R1's self-judged label and the Sonnet R1 column beside it.
- Prediction 11's agreement half (0.85 to 0.93 overall, lower on the 8B's answers) is scored on the random subsample
  only: overall and per-generator agreement of the final grade() flags, Sonnet vs qwen, with Wilson 95% intervals and
  n. Agreement on the discordant set is reported but never compared with that band.
- In the public verdict summary, the agreement fields named "overall" and "per_generator" cover the discordant rows
  only under this decision; the record text relabels them and they are never read as unselected agreement.
- Any judge-error bound that the preregistration or amendment 2 takes from overall primary-vs-sensitivity
  disagreement (the epsilon bound) uses the subsample's disagreement rate, never the discordant set. The noise-floor
  item "judge disagreement on discordant pairs" stays on the discordant pairs, as written.

IF SONNET BECOMES UNAVAILABLE
Unavailable means the alias is no longer listed, its upstream cannot be confirmed as Sonnet 4.5 (check above), or the
model is retired. An outage or the $3/day cap does not count; judging resumes from the cache.
- Partway through SELECT: the row is published on the pairs already judged. It states how many were and were not
  judged, and it is labelled incomplete and non-random (judged in question-id order); pair-level statistics use only
  pairs with both answers judged.
- At CONFIRM: the CONFIRM co-primary row, including the Sonnet-substituted G1 check, is reported as unavailable.
- In either case no other model replaces Sonnet. The primary verdicts and all gates use qwen and are unaffected.
- Amendment 2 does not wait for this row. If the SELECT row is incomplete when amendment 2 is filed, it is reported
  there with n judged and n not judged, and completed later.

EXPLORATORY: REFERENCE-BIAS SUBSAMPLE (SELECT only, never a gate)
The discordant pairs are chosen by qwen's own disagreement between arms, so they cannot estimate an arm-dependent
judge bias. A random subsample can.
- Draw: the 60 SELECT questions with the smallest sha256("ks-cascade-refbias-2026-11-08:" + question id), with both
  generators' answers on each ("60 per generator", paired by question). That is at most 120 unique answers, and a text
  both arms share is judged once. Unstratified (by question type); the hash ranking is the only randomisation. No
  verdict is read to draw the sample.
- Both judges grade every sampled answer, whether or not it is discordant. Sonnet's subsample verdicts go to their own
  cache, separate from the discordant row; the qwen verdicts are the primary's.
- Estimand: b = E[J_sonnet - J_qwen | A1] - E[J_sonnet - J_qwen | A0], with J the final grade() flag, paired by
  question, on the accuracy scale. The 95% interval is an unstratified question-level paired percentile bootstrap
  (10,000 resamples, seed 0). Reported alongside: answer length in characters per generator, the number of questions
  with d(A1) different from d(A0), and the number of sampled questions that contribute zero by construction (identical
  texts, abstention overrides). If fewer than 5 questions have d(A1) different from d(A0), the interval is reported
  but the prediction is scored as not tested; a [0,0] interval does not exclude 0.
- Application: b is applied to CONFIRM decisions under the assumption that the bias is the same across splits. tau*
  and delta (SELECT, judge-derived) are reported with b beside them but are not relabelled.
- Reporting rule: it covers the accuracy-scale gated decisions only, each with w = the difference between the two
  compared arms in the share of questions answered by A1: G1's accuracy half (A1 vs A0: w = 1; a cascade or A3-0 vs
  A0: w = its r), G2's dAcc(cascade - A1) (w = 1 - r), G2's dAcc(cascade - A3-0) (w = r minus A3-0's r) and G3's
  dAcc(A4 - A3) (w = the difference of their r). G2's routing gain (w = 0 by construction), G4 (its accuracy-loss term
  depends on the absolute judge bias, not on b), and every cost, AUROC and H1 decision are not covered. For each
  covered decision let m = |bound - bar| on CONFIRM, whether it passed or failed. The decision is reported as
  judge-dependent, with b and its interval beside it, if |w| x |b_hat| >= m/4, or if its pass/fail outcome flips when
  the statistic is shifted by w times either endpoint of b's interval. At n = 60 the interval half-width is expected to
  be about 0.1, so the second clause is expected to flag most decisions; that is stated now as a limit of the sample
  size, not as evidence.
- Prediction, written now: the interval excludes 0 with b > 0, meaning Sonnet is relatively more lenient than qwen on
  the 8B's answers. Prior 40%.
- Cost: about 120 Sonnet calls at about $0.003, about $0.40, within the daily cap. If Sonnet becomes unavailable
  first, b is reported on the questions whose every sampled answer has a verdict from both judges (with that n), or
  as not estimated if fewer than 30 such questions exist.

PUBLISHED
Aggregates only: agreement counts and rates, b and its interval, the G1 side-by-side bounds, answer lengths and
Sonnet dollars. No question ids, answers or prompts.

Written by Claude Code in a session run by the maintainer, who made this decision. The reference-bias question is
Kannaka's, from her rStation reflection of 2026-11-08 14:39Z.

content hash 1a1d3271660012fd86a47e399ae96c4b49b97458d178d3eb7377c13a97c25109

Machine check: claimed

Derived from verify_record.py at b3ba75d; it never changes the standing.