nodwyn.
Menu

ADR 0047: A task's judge is a choice, and Settings sets judges up

  • Status: accepted; implemented 2026-10-08
  • Date: 2026-10-08
  • Amends: 46 (the judge is chosen per task; a plan's sign-in is no judge; its model is not typed), 41 (Settings is tabs; providers carry their models), 30 (a paired device chooses no judge)

Context

ADR 46 made the judge a provider's: [judge] server names a Jev server or an OpenAI one, and Settings › Judge chose between them. Using it showed three faults, each in the logs of the person's own sessions:

  • An OpenAI judge did not work. Settings offered every OpenAI server at api.openai.com, a ChatGPT plan's among them. The keychain sends a plan's sign-in only to /v1/models and /v1/responses (keychain openai.go), so every judge question on it failed with that refusal: three jev.req records on 2026-10-07 say so. ADR 46 had left that open ("the first answer would show"); it showed.
  • Setting up Jev took knowing jev-latest. The judge's model was a free text field, and the server's name, Jev, typed there sent "model": "Jev" to typesafe.ai, which answered 400 Bad Request (session 96faa033).
  • Its price had to be typed. nodwyn knows a judge's price by its host and model (catalogue.JudgePrice), so with the model wrong there was no price, and the person typed 0.042 into [judge].

The person also asked for the judge to be chosen beside the model, for each task as for the default, and for Settings to have a Judge tab of its own, with the model settings revamped together.

Decision

  1. A task's judge is a choice. core.Options.Judge opens a task with a judge of its own: a [servers.NAME] whose provider answers typed questions, or none; left out, it is [judge]'s, as before. Session.SetJudge moves it between turns, refused while a turn runs or an attachment is added: a judge chosen mid-turn would judge half of it. Each choice is a session.judge record (server, provider, host, model, actor) written before the change, so the questions asked after it come after it in the log. Resuming restores the last record's judge; one config.toml no longer offers leaves the task with none, with a note saying why, never with [judge]'s, which would send its plans and commands somewhere nobody chose for it.
  2. The judge's primitives read the task's judge on every call. They are registered when a judge could be asked at all (a keychain open, judge.content not none, config.toml naming a judge provider), since the primitive table is fixed for the session (ADR 9) while the judge is not. Each call asks the session (prim.JudgeClient.JudgeOptions) for the judge it has then, and fails, as no answer, when it has none. Their egress metadata, the host and key class the guard checks each call against, is moved to the new judge while the task is held idle (prim.RetargetJudge), so the guard never judges one destination while bytes go to another. judge-available? (hidden) says whether the task has a judge whose key is stored, which the plan gate asks. The judge's tools are offered to the model only at judge.content = "full" and only when the task starts with a judge it can ask: the tool list cannot gain them later.
  3. A ChatGPT plan's sign-in is no judge. core.CheckJudge refuses it when a task opens with it or moves to it, and settings.write refuses it as judge.server, in a sentence that says the Decisions API takes an API key. settings.read's new judges lists it with state plan, so the console can say what to add instead.
  4. The judge's model is not typed. A judge asks its provider's judge model (config.JudgeModels: jev-latest, gpt-6-luna), which is all either API answers to, so Settings no longer has a field for it. [judge] model and [judge] price_* stay for a hand-written override and apply only on the server [judge] names, since they were set for it: a task that chooses another judge is not asked or priced as that one (config.JudgeOn). Settings says when [judge] model names a model its provider does not answer to, and puts it right in one click (clearing it). A price is offered only under a fold, as an override.
  5. Settings is tabs. General (approvals, limits), Models, Judge, Context, Documents & containers, Keys & devices and This browser, each at #/settings/<tab>, so a link opens one and a reload keeps it; only the tab shown is drawn, and the save bar keeps edits from every tab. - Judge says what new tasks ask, chooses the default among the judges set up, and sets one up in place: Jev with its key alone, or an OpenAI API key, each with what it is and why nothing else is asked. A judge added when there is none becomes the default. Each judge is a card with its host, model, price and state; a Jev one's key is replaced and the provider removed there, since it carries no model. - Models lists each provider as a card with its models under it, with Add a model there, and the default for new tasks as Model · Effort · Judge. A Jev server is no longer a model provider's preset or kind: it is a judge, under Judge. A provider form that needs a key while the keychain is locked has the keychain's form beside it.
  6. The judge is chosen beside the model. The model picker has a Judge select wherever it is: New task (sent as session.open's judge when it differs from the default), a conversation (session.judge.set, which waits for a running response to end) and Settings' default (judge.server). A judge that cannot be asked does not stop a task, since the person answers instead; the picker says why it cannot.
  7. A paired device chooses no judge. The judge is shown a task's plan and commands, so where they go is this computer's choice, as a model's is (ADR 30, ADR 40): the bridge lists session.judge.set only for this computer, and the daemon refuses judge in session.open and session.judge.set from a device.
  8. The terminal client chooses too. :judge shows the task's judge and the judges set up; :judge NAME and :judge none move it.

Consequences

  • An OpenAI judge works with an OpenAI API key, which Settings › Judge adds in one step; a person with only a ChatGPT plan is told what to add, not left with a judge that refuses every question.
  • Setting Jev up takes its key, nothing else. A config that already says model = "Jev" is shown as wrong with its fix.
  • Two tasks may ask different judges at once, and the log of each says which. [judge] is the default, no longer the only judge.
  • The judge's primitives exist in more sessions than before (whenever a judge could be asked), and fail when the task has none; (plan-judge?) asks judge-available? rather than whether they exist.

Not built

  • Choosing a judge while a turn runs, queued for its end as a model is: it is refused, and the console's select waits.
  • A judge for nodwyn run and line mode other than [judge]'s, and a --judge-server flag.
  • Asking OpenAI's Decisions API for real: the tests ask a fake that speaks it as its guide documents. A person's first question with an API key is still the first evidence that it answers.

Amendment 2026-10-16: Settings is five tabs, saves one way, and has one place for each thing

Settings had grown seven tabs, three ways of saving and two places to set the default judge, so that a person could not tell what a click had saved. Now:

  • One rule for saving. A choice made in Settings waits in the bar at the foot of the page and is saved with the others: every settings.write setting, the default model and effort (settings.control.write) and the answer for uploads to Google Docs (settings.docs.write), each written once the bar saves and forgotten once it has. What acts at once is a key or the passphrase (a secret is never staged), a provider or a model (each its own form with its own Save) and the choices of This browser, which change nothing on the computer. The page says so once, at its top.
  • Five tabs. Context, and Documents & containers, each held one rarely used section; they are folded into General under an Advanced fold (the limits, the context window, the container runtime), and the Google Docs answer is an approval. #/settings/context and #/settings/tools open the fold and are replaced by #/settings/general. The transport exception (policy.allow_insecure_http) is a key matter and moved to Keys & devices.
  • The default judge is the Judge tab's alone. The Models tab's default row is a model and an effort, and links to it. Judge off has one way, No judge: the reserved plan+summaries level and the Nothing (off) chip are not offered (a hand-written value is shown until a level is chosen), and the threshold, timeout and prices are a Tuning fold.
  • One place for a key's value. Credentials changes it; a provider's form and a judge's card link there. A ChatGPT plan is connected on its provider under Models only: Connect an account offers no openai.
  • Paths in words. A reference to Settings names a tab (and, under General, a section): TestSettingsPathsNameATabAndSection reads the docs and the console's own strings for it.

Amendment 2026-10-17: the person's device chooses (ADR 30)

§7 holds for a device paired by a plain invitation. A device signed in with the person's passkey chooses a task's judge, at open and between turns, as this computer does (ADR 30, amended 2026-10-17): the passkey says it is the person, which is what the choice was kept to this computer for.

Amendment 2026-10-28: a task opened with no judge set up can still choose one

§2 registered the judge's primitives only when config.toml already named a judge provider as the task opened, and a task kept the config it opened with, so a person who added Jev or an OpenAI key afterwards saw it in the task's Judge select and was told to close and resume the task. Now the primitives are registered whenever a keychain is open and judge.content is not none, and session.judge.set hands the task the daemon's current [servers] and [judge] tables when a choice succeeds (Session.SetJudgeFrom). Only those tables are taken: the task's model and window stay what it opened with. The judge's tools offered to the model still depend on the judge the task started with, since the tool list cannot gain them later. A task opened with no keychain, or with judge.content = "none", still has to be resumed.

Amendment 2026-10-27: a refused judge choice changes nothing

The review found that installing the daemon's judge configuration before checking the choice let an invalid or busy request change the task's current judge without a session.judge record. SetJudgeFrom now holds the idle session, validates the candidate configuration and choice, records the successful choice, then installs both together and retargets the primitives. TestRejectedJudgeChoiceKeepsTheTasksConfiguration exercises an invalid choice and a request made while the model is answering.