Earned Autonomy — a three-tier router that demotes work to your own GPU

You are an ML platform engineer. Build me an earned-autonomy router with four tiers: frontier (Fable 5 via the API), mid (Opus via the API, or GPT-5.6 via the Codex CLI), local (Qwen 27B Q4 on my 24GB GPU via llama-server) and zero (plain shell, no model).
1. Log every delegated task for 14 days in SQLite with the class, tier, tokens, cost, stop_reason and the reviewer's verdict. Treat stop_reason 'refusal' or 'max_tokens' as a fail even on HTTP 200, as @raycfu warns.
2. Run the local tier in shadow mode on two low-stakes classes. Strip tier labels and have a reviewer from a different vendor grade blind. Double-grade 10% and report kappa.
3. Promotion is not '20 in a row'. Use SPRT with p0=0.85 and p1=0.95, plus a Wilson bound, and demote when the upper bound falls below 0.90.
4. Rerun 50 Terminal-Bench tasks locally to check Qwen's self-reported scores.
5. Reconcile the cost ledger against billing exports, including review cost and local power.
6. Use API keys only. No subscription OAuth tokens in third-party tools, and never print key values.
Report per class: tier, n, pass rate with CI, dollars saved, and whether the falsifier fired.

ROLE
You are an ML platform engineer. You build measurement before automation and treat vendor benchmarks and creator claims as hypotheses.

CONTEXT
- @thomas.lentine: Fable orchestrates, writes specs, spawns cheaper Opus subagents in worktrees, and reviews in a loop.
- @raycfu: a cron loop.sh in which a cheap model triages, Fable plans, Sonnet executes and a fresh Fable reviews. Tasks earn autonomy 'after 20 passes at 95%' and lose it 'below 90%'. Refusals 'come back as HTTP 200', so check stop_reason.
- @cooper.simson: a /route skill that sends execution to GPT-5.6 through the Codex plugin. @nateherkai: Claudex, where two vendors cross-audit plan and build.
- @gigaqian: Qwen 3.8 27B fits in about 17GB at 4-bit on a 3090/4090 and claims 84.3 on OSWorld, but these are 'Qwen's evaluations'. DeepSeek V4 Flash went from 61.8 to 82.7 on Terminal Bench 2.1 through re-post-training. The Xiaomi AI Cube is an 80GB unified-memory prototype.
- @techno.optimist.prime: Friedberg says token share flipped from 80/20 closed to 80/20 open in 12 weeks.
- @tech.unicorn: Anthropic blocked subscription tokens in third-party tools, an estimated jump from $200 a month to $500-$2,000. @duck.tate: plan-value estimates of $7,000 and $12,000 in usage, which are third-party figures.
- @githubsignals: only-cli/oc gives compact numbered page views. @shellonback: never send an LLM a command you could run in bash.

HARD CONSTRAINTS
- Comply with vendor terms. Automation uses API keys. No subscription OAuth tokens in third-party harnesses. Ignore the 'don't tell Anthropic' workaround.
- Never show or log secret values. Only check whether a key exists.
- The local tier stays in shadow mode until promoted: it runs, but the incumbent output ships.
- Promotion uses statistics, not streaks. With 20 of 20 passes, the 95% upper bound on the fail rate is still about 14%.
- Graders never see which tier produced an output.

DELIVERABLES
1. router/ containing: classify.py (rules first, a cheap model as fallback); tiers.yaml; zero_tier.py, which maps requests to commands in package.json, the Makefile or scripts/; and stop_reason handling.
2. The ledger.sqlite schema and 14 days of logged tasks.
3. The local tier: llama-server with a Qwen 27B Q4_K_M GGUF, plus llama-bench output (tokens per second and the largest context that fits).
4. A blind grading harness: labels stripped, a cross-vendor reviewer, and a 10% double-graded sample for Cohen's kappa.
5. promote.py implementing SPRT per class and tier, with Wilson-bound demotion.
6. An independent rerun of 50 Terminal-Bench tasks on the local model, compared with the vendor numbers.
7. A cost report reconciled to billing, covering review spend and GPU power, with dollars saved per class.

BUILT-IN GAP ANALYSIS
List the classes with fewer than 30 graded tasks, reviewer disagreement above 20%, local runs with truncated context, and any cost that was estimated rather than billed. Fill what you can, by adding tasks or a second reviewer. Mark the rest UNFILLED.

SELF-QUESTIONS
Does my reviewer favor its own vendor? Does the local tier fail on long-context tasks the classifier hides? Are refusals counted as passes? Would the zero tier have handled 10% or more of the 'tasks'?

STOP
Stop at day 14 with a verdict per class. Stop immediately and ask if any harness needs subscription tokens or a terms grey zone.

RETURN
Return router/, ledger.sqlite, grading/, bench/, costs.csv and report.md with the sections Setup, Classes, Promotion Results, Independent Benchmarks, Costs, Falsifier, Compliance and Gaps.

Collect the raw material before writing any router code.
(1) Export 30 days of billing from the Anthropic and OpenAI consoles as CSV, with dollars and tokens by model.
(2) Pull the last 200 delegated tasks from the orchestrator's history: agent branches, task files or transcripts. Hand-label each with a class: rename, test-writing, data cleanup, UI tweak, docs, refactor or research.
(3) Quote the Anthropic Consumer Terms and Usage Policy clause on subscription credentials in third-party tools that made @tech.unicorn's setup non-compliant. Quote the equivalent OpenAI Codex terms.
(4) Build claims.csv for Qwen's 84.3 on OSWorld, the 17GB quant size, DeepSeek's 61.8 to 82.7, Friedberg's 80/20 flip, and @duck.tate's $7,000 and $12,000 plan values. Mark each one vendor-reported, third-party or independently verified.
(5) Read the Messages API docs and list every stop_reason value.

Write router/ in Python.
classify.py applies regex and AST rules first: a 'rename' touches identifiers only, and a 'test-writing' change only adds test files. A cheap model is the fallback and returns a class and a confidence score.
zero_tier.py parses package.json scripts, Makefile targets and scripts/*.sh. When a request maps to an existing command, such as 'run the tests' or 'build', it runs the command directly and logs tier=zero and cost=0, as in @shellonback's detector.
tiers.yaml maps tier names to endpoints. Frontier is Fable 5 via the API. Mid is Opus via the API or GPT-5.6 via the Codex CLI. Local is http://localhost:8080/v1.
Every model call records stop_reason. 'refusal' or 'max_tokens' marks the task failed.
The ledger table is tasks(id, ts, class, tier, shadow, model, in_tokens, out_tokens, cost_usd, latency_s, stop_reason, tests_pass, reviewer, verdict).

Download a Qwen 27B Q4_K_M GGUF, or the Unsloth dynamic 4-bit quant, from Hugging Face and record its SHA-256.
Run llama-bench to get prompt and generation tokens per second. Find the largest context that fits in 24GB with the KV cache at q8_0.
Serve it with: llama-server -m <gguf> -c <ctx> -ngl 99 --port 8080.
Run 50 Terminal-Bench and 50 SWE-bench Verified tasks through the same agent harness and tool access the router uses, with oc for web reads. Run a frontier baseline on the same 100 tasks.
Compare the pass rates with Qwen's self-reported figures.
Record where the model drops reasoning across tool calls, and how much it degrades past 32k tokens. Log GPU memory headroom at the chosen context, so you know how much room is left for longer agent sessions. Note which failure types are tool-format errors rather than reasoning errors, because a better system prompt or grammar constraint can fix the former.

Pick two classes with low blast radius and executable checks, such as test-writing and data cleanup.
For 14 days, send every task in those classes to both the incumbent and the local tier. Only the incumbent output ships.
The grading harness strips model names, tier labels, signature comments and whitespace tells, then shuffles the order.
The primary reviewer comes from a different vendor than the producer. This is @nateherkai's Claudex rule: never let one model plan, build and grade its own work.
A second reviewer double-grades 10% of the tasks, and you compute Cohen's kappa.
The verdict requires both tests_pass (exit code) and the reviewer's pass against the spec. Log rows with shadow=1. Seed the shuffle and store the mapping from blind ID to tier in a separate file the reviewers never see. Record reviewer cost per graded task, because step 6 needs it.

Implement promote.py.
For each class and tier, run Wald's SPRT on the pass-or-fail sequence with H0 p=0.85 and H1 p=0.95, alpha=0.05 and beta=0.10. Accepting H1 promotes the class one tier down.
Compute the Wilson 95% interval in parallel. Demote immediately when its upper bound falls below 0.90, which is @raycfu's 'below 90%' rule in statistical form.
Document why a streak is weak: 0 fails in 20 leaves a 95% upper bound of about 0.14 on the fail rate.
Promotion is per class, never global. Promoted classes keep a 10% audit sample graded by the frontier reviewer.
Add goals/, @raycfu's daily regression suite, which replays 10 frozen tasks per promoted class every morning via cron. Store every decision with its timestamp and the evidence count, so the report can show exactly when and why each class moved. Expose the thresholds in tiers.yaml rather than hard-coding them.

For every task row, compute the cost from metered tokens at current list prices.
For the local tier, compute power as watts from nvidia-smi --query-gpu=power.draw, times latency, times the local $/kWh, and add hardware amortization over 36 months.
Reconcile the router's frontier and mid cost against the billing exports for the same days, to within 5%.
Per class, compute the counterfactual: the cost if every task ran on frontier, minus actual cost.
Include review spend. Reviews can erase the savings on cheap classes.
Treat @duck.tate's plan values and @tech.unicorn's $500-$2,000 estimate as claims to compare against, never as inputs. Store every price used, with a source URL and date, in prices.yaml so the numbers can be recomputed when list prices change. Show per-class savings both with and without review cost.

Run the oracle pass:
(a) Per class, the local pass rate with a Wilson CI against the incumbent, on executable tests.
(b) The independent benchmark reruns against the vendor-reported numbers.
(c) The billing reconciliation within 5%.
(d) Reviewer kappa of 0.6 or higher.
(e) The falsifier: does any class sustain a Wilson lower bound of 0.90 or higher on the local tier?
Write report.md with the sections Setup, Classes, Promotion Results, Independent Benchmarks, Costs, Falsifier, Compliance and Gaps.
If the falsifier fires, say plainly that you stay two-tier because the local floor is not there yet for this work. Rank the gaps by dollars at stake. Include one row per class showing what would change the verdict, for example 'needs 40 more graded tasks' or 'kappa too low to trust'. Recommend the next class to put in shadow mode.

You are an independent reviewer. Open report.md, ledger.sqlite, promote.py, grading/, bench/ and costs.csv.
1. Enumerate every claim: pass rates, promotions, benchmark comparisons, savings and compliance. Mark each one verified (with a SQL query or file reference), unverified, or false.
2. Recompute the per-class pass rates and Wilson intervals from ledger.sqlite and diff them against the report.
3. Rerun promote.py on the ledger and check that the promotions match.
4. Check blindness: grep the grader prompts for model and tier names, and recompute kappa.
5. Check refusals: count rows with stop_reason 'refusal' and confirm none is marked pass.
6. Check that savings include review spend and local power, and that they reconcile to billing.
7. Check compliance: no subscription OAuth tokens in configs, and no key-shaped strings in logs. Search by pattern and report counts without printing any match.
8. Rank the issues by impact. Fix the top three now. Report what you could not fix and why.
Return audit.md with the columns claim, status, evidence and fix, plus 'Remaining gaps' and an A-F grade.