Source confidence: high
See also: Deeper Dive — Skills That Earn Their Lines — benchmark-gated self-editing plus subtraction →

Skills That Earn Their Lines — benchmark-gated self-editing plus subtraction

Insight class: Workflow Multiplier · Difficulty: Medium · Quality: 5/5 Connected entries: 11 · Tools: Anthropic, Antidoom, ChatGPT, Claude, Claude Code, Grokbot, OpenAI, Prime Agent, SkillOpt, Soup

In short: Build a two-way, eval-gated ratchet for your CLAUDE.md and skills: a held-out eval from your own history, an ablation sweep that deletes rules whose removal does not hurt, and a SkillOpt-style loop that merges only edits beating the benchmark - with a noise check that proves the eval is signal, not variance.

Key fact Detail
What it covers agents, prompt engineering, LLM, open source
Tools Claude, Claude Code, Anthropic, ChatGPT, Antidoom
Posts covered 11 posts (2 S-tier, 4 A-tier) · top: Claude Code simple mode — ablating system… · Microsoft SkillOpt — agent skills that rewrite… · Soup — fine-tune LLMs on a 4GB laptop GPU with…
Creators @angus.sewell, @raycfu, @cooper.simson
Sources dated Jun 2026 – Oct 2026
Difficulty Medium
Prompt system Run-it-yourself prompts
Last verified 2026-10-22

Thesis

Successor to the harness and second-brain dives. Those stop at 'append a lesson after every mistake'. The catalog now holds the missing machinery — Microsoft SkillOpt rewrites skills only when a benchmark improves, bilevel autoresearch adds a coach loop over the worker, Prime Agent rewrites its own harness to 95.5% on ARC-AGI-3 — and, crucially, evidence that subtraction matters as much as addition: Boris Cherny's simple mode and the setup-purge post show stale rules make newer models dumber. Combine them into a ratchet that both adds and deletes rules, keeping only edits that beat a held-out eval.

Why it matters

Every power user's CLAUDE.md grows monotonically until it is a tax on every call. A two-way, eval-gated ratchet turns rule accretion into measured improvement, and the winning traces become fine-tuning data you can push into a small local model with Soup.

The evidence — connected entries

Microsoft SkillOpt — agent skills that rewrite themselves against benchmarks

Creator: @marc.kaz · Tier: S Tools: SkillOpt

Core mechanism — skill rewrites accepted only when a benchmark improves

Microsoft open-sourced SkillOpt: a base model runs a task, an optimizer rewrites the skill instructions from results, successful paths are separated from failures, and edits that don't beat the benchmark are auto-rejected. Claimed to beat hand-written prompts and TextGrad and to transfer across models.

View original post

Creator: @raycfu · Tier: A Tools: OpenAI

Outer loop — a coach that rewrites the search playbook, not just the answer

Builds on Karpathy's autoresearch loop (try change, test, keep or revert). An outer coach loop watches the inner worker and rewrites its search playbook so it stops repeating the same ideas; claimed ~5x better on Karpathy's benchmark with the same models.

View original post

Prime Agent — self-rewriting harness lifts Opus 5 to 95.5% on ARC-AGI-3

Creator: @wassimyounes_ · Tier: A Tools: Prime Agent

Proof of ceiling — a self-rewriting harness beats the bare frontier model

Prime Intellect released Prime Agent, an open-source coding harness built on a Recursive Language Model (RLM) framework that rewrites its own prompts, memory, tasks and scaffolding at runtime based on failures. Opus 5 alone scores 30% on ARC-AGI-3; with the harness, 95.5%, a hair over the 95.4% human baseline. The real story is test-time compute and self-modifying scaffolding.

View original post

Self-improving skills vs AI teammates — log issues, rewrite skills

Creator: @angus.sewell · Tier: B Tools: Grokbot

Signal source — skills log their own failures into a table as they run

Argues personified 'AI teammates' are static while skills are adaptable: keep every skill in a database table, have each skill log issues about itself when used into a second table, then periodically run a prompt that reads the accumulated issues and rewrites the skill. Specific workflows with clear end states beat persona prompts.

View original post

Systems that prompt your AI — DB-driven loops with skills and cron

Creator: @angus.sewell · Tier: B Tools: Claude, Claude Code

Scheduler — DB-driven cron loops that prompt the agent instead of you

Riffs on Boris Cherny's advice to build loops that prompt Claude instead of prompting it yourself: connect the AI to a database with work and issues tables, use cron jobs to detect pending changes, and run skills for scoping, implementing, testing and repeating until the human goal is met. Counterpoint: for go-to-market work the human is still the bottleneck and must stay in the loop.

View original post

Chat-history mining — derive better custom instructions from your AI logs

Creator: @angus.sewell · Tier: B Tools: ChatGPT, Claude

Eval mining — your exported chat history becomes the held-out test set

Export your full ChatGPT or Claude history, have a small model like Jev tag every conversation for mistakes and usage patterns, then feed the analysis to your main assistant to write new custom instructions. A cheaper alternative to building a full personal 'Jarvis' agent.

View original post

Claude Code simple mode — ablating system prompts makes the model smarter

Creator: @foundedceo · Tier: S Tools: Anthropic, Claude, Claude Code

Subtraction evidence — stripping the system prompt makes the model smarter

Boris Cherny at YC Startup School: Claude Code accepts --system-prompt for a custom system prompt, and an undocumented simple mode (env var CLAUDE_CODE_SIMPLE=1) deletes all system and tool prompts. Anthropic uses it as an ablation and finds the model is slightly more intelligent without them, though product prompts still help it behave as users expect.

View original post

Claude setup purge — audit stale CLAUDE.md rules for newer models

Creator: @cooper.simson · Tier: B Tools: Anthropic, Claude, Claude Code

Subtraction practice — rules written for old models now hold new ones back

Argues rules and skills written for older Claude models now hold newer ones back, citing Anthropic cutting much of its system prompt after Opus 5. Recommends a prompt that audits every CLAUDE.md rule and skill, deletes dead weight and rebuilds a lean setup; the prompt itself is comment-gated.

View original post

Liquid AI Antidoom — FTPO training to kill reasoning doom loops

Creator: @gigaqian · Tier: A Tools: Antidoom, Claude

Failure-specific training — target the exact loop token, a template for targeted fixes

Antidoom is Liquid AI's open-source pipeline that finds the exact token restarting a repetition loop, marks it rejected, picks up to 20 alternatives, and trains with Final Token Preference Optimization (FTPO) plus a KL-like regularizer in logit space so reasoning isn't damaged.

View original post

Soup — fine-tune LLMs on a 4GB laptop GPU with one command

Creator: @martintrainhard · Tier: A Tools: ChatGPT, Claude, Soup

Distill step — fine-tune a small local model on winning traces with a 4GB GPU

Soup, built by a 17-year-old solo founder from Kazakhstan, wraps LLM fine-tuning into a simple flow: describe the goal, point at your data, run soup train, and it auto-picks batch size and memory settings. Claimed 85k users training medical assistants and reasoning models on consumer GPUs. Link gated; verify the claims.

View original post

Self-improving trading agent — critic loop that rewrites losing strategies

Creator: @raycfu · Tier: B

Cross-domain instance — a critic loop rewriting losing trading strategies, same shape

Workflow pitch: describe a strategy, an agent codes it and wires the broker API, runs a five-year backtest, and a critic agent dissects losing trades and rewrites entry/exit logic before rerunning. Hard-coded stop-loss and drawdown limits sit outside the agent's reach, and failed strategies go to a local DB the agent learns from. Guide is comment-gated.

View original post

Action sketch — how to implement this

Export your last 3 months of Claude history, have a small model tag 40 representative tasks with pass/fail criteria — that is your held-out eval Run an ablation sweep: drop each CLAUDE.md rule and skill one at a time, rerun the eval, delete anything whose removal does not hurt the score Add the SkillOpt loop on a nightly cron — skills log issues, an optimizer proposes edits, only score-improving edits merge; export passing traces for a Soup fine-tune later Falsifier: if eval scores vary more between two identical runs than between best and worst rule sets, your eval is too noisy and the whole ratchet is optimizing noise

Prerequisites

Key tools and resources

Creators to follow

Prompt system — run this yourself

Everything below is pasteable into Claude Code, Claude, or any agent. Use the one-shot when you want the whole result in one run; use the step prompts when you want to steer. Every step carries its own audit, and the gap audit at the end makes the agent find and fill what it missed.

Mission. Build a two-way, eval-gated ratchet for your CLAUDE.md and skills: a held-out eval from your own history, an ablation sweep that deletes rules whose removal does not hurt, and a SkillOpt-style loop that merges only edits beating the benchmark - with a noise check that proves the eval is signal, not variance. Done means rules are both added and removed only on measured gains, winning traces are exported for a possible local fine-tune, and the ratchet is provably optimizing signal rather than noise.

1 · Deeper, not bigger

Before running anything, make the task deeper rather than larger. Ask: what would a top professional do here that I am not asking for; what is the rawest material; which methods and tools; what, other than the model or me, proves it right. Then ask for exactly that.

Rewritten prompt — ask for exactly that:

Act as an agent-optimization engineer. Build me a two-way, eval-gated ratchet for my CLAUDE.md and skills - one that DELETES as readily as it adds, and only on measured gains. Start by having a cheap/local model tag ~40 representative tasks from my exported chat history with objective pass/fail criteria; that is my held-out eval. Check noise first: run the same setup three times and measure variance - if variance rivals the gains I expect, tell me the eval is too noisy before I build anything on it. Then run an ablation sweep: drop each CLAUDE.md rule and each skill one at a time, re-run the eval, and delete anything whose removal does not hurt the score (stale rules for old models drag new ones down - cooper.simson, Cherny's simple mode). Then add a SkillOpt-style loop (github.com/microsoft/SkillOpt): skills log failure notes, an optimizer proposes edits, and only edits that beat the benchmark merge; put it on a nightly cron. Add a bilevel outer loop (raycfu) that revises the optimizer's strategy, not just the text. Lock my safety/risk rules outside the editable set. Export passing traces for a possible Soup fine-tune. Return the eval, the noise check, the ablation deletions and keeps, the gated edits, and a clear verdict on whether the ratchet beats baseline beyond noise.

2 · One-shot prompt

You are an agent-optimization engineer building a two-way, eval-gated ratchet for a CLAUDE.md and skills folder. Stance: subtract as readily as you add; nothing changes without a measured gain beyond noise; safety rules are never editable by the loop. No marketing language.

Context and method: appending a lesson after every mistake only grows the prompt into a tax. Microsoft SkillOpt (github.com/microsoft/SkillOpt) treats skill text as trainable parameters, gating every rewrite on beating a benchmark and auto-rejecting edits that do not. Boris Cherny's simple mode (foundedceo) and cooper.simson's purge show stale rules written for older models make newer ones dumber - so you must ablate and delete. raycfu's bilevel loop adds an outer loop that revises the optimizer's search strategy, and his trading pattern locks risk limits in code the loop cannot touch. wassimyounes_'s Prime Agent caveat is load-bearing: a 0.1% margin is noise, not improvement. angus.sewell: exported chat history is the best personalization dataset.

Hard constraints: build the held-out eval BEFORE any editing; measure the noise floor and answer the dive's falsifier; ablate every rule/skill; every merged edit must beat the benchmark beyond variance; safety/risk rules live outside the editable set. Name real tools (SkillOpt, Soup), real files, real paths. Do not claim a gain inside the noise band.

Ordered deliverables: (1) the held-out eval - a cheap/local model tags ~40 tasks from exported chat history with objective pass/fail criteria. (2) a noise-floor check: variance of 2-3 identical runs vs the gains you expect; a go/no-go on trusting the eval. (3) an ablation sweep: drop each CLAUDE.md rule and each skill one at a time, re-run, delete what removal does not hurt (never deleting safety/money rules). (4) a SkillOpt-style loop: skills log failure notes, an optimizer proposes edits, only benchmark-beating edits merge; on a nightly cron. (5) a bilevel outer loop revising the optimizer's strategy. (6) safety/risk rules locked outside the editable set. (7) an export of passing traces for a possible Soup fine-tune.

Built-in gap analysis: before finishing, list what is unverified - are the eval criteria objective; is any reported gain inside the noise band; did the ablation actually delete anything; are safety rules truly outside the editable set; did any merged edit fail to beat baseline. Fill what you can.

Self-questions to answer in writing: Did I measure noise before trusting the eval (the falsifier)? Did the ratchet DELETE rules, or only add? Is any gain smaller than run-to-run variance? Are safety/risk rules genuinely uneditable by the loop? Did the optimizer merge an edit that did not beat the benchmark? Does the bilevel loop change strategy or just re-run the same instincts? Are the exported traces actually the passing ones?

Stop conditions: stop and ask only if there is no exported history to build an eval from. Otherwise proceed end to end.

Return format: a markdown report with the eval description, the noise check and falsifier verdict, the ablation table (rule/skill, score with/without, deleted/kept), the gated-edit log (edit, before/after score, merged/rejected), the locked safety set, and an 'Unverified and missing' section. Commit the eval, the ablation/optimizer harness, and the ratcheted CLAUDE.md/skills; print the hash.

3 · Step-by-step prompts

Step 1 — Build the held-out eval from history

Goal: Turn exported chat history into an objective eval with a cheap model.

Establish the oracle. Export the chat history (angus.sewell's point: it is the best personalization dataset). Have a cheap or local model (Haiku-class) tag ~40 representative tasks from it, each with an objective pass/fail criterion a human could apply (not 'good' - e.g. 'produced valid JSON', 'followed the plan-first rule', 'refused unsafe action'). Make this a held-out set never used to tune rules. Build a runner that scores a given CLAUDE.md+skills setup against all ~40 tasks. This eval is what every ablation and every optimizer edit is judged against; a change that does not move it does not merge.

Expect: ~40 held-out tasks tagged with objective pass/fail criteria, plus a runner that scores a setup against them.

Audit before moving on: Check 5 criteria are objective enough for two graders to agree, and that the tasks span real usage (not just easy ones). Fail if criteria are subjective or the set is cherry-picked. On fail, re-tag with tighter criteria.

Step 2 — Noise floor + falsifier

Goal: Prove the eval is signal before building the ratchet on it.

Answer the dive's falsifier before anything else. Run the SAME setup against the eval 2-3 times and measure the variance between runs - this is the noise floor. Then estimate the gap you expect between the best and worst rule sets. If run-to-run variance is comparable to or larger than that best-vs-worst gap, STOP and report that the eval is too noisy and the whole ratchet would be optimizing noise; enlarge the eval or tighten criteria until variance is clearly below the gains you need to detect. Only proceed past this step if the noise floor is small enough to trust.

Expect: A measured run-to-run variance, a stated expected best-vs-worst gap, and a go/no-go verdict on the eval's trustworthiness.

Audit before moving on: Confirm the repeated runs used identical config. Fail if you proceed while variance rivals the expected gains. On fail, enlarge/tighten the eval before building the ratchet.

Step 3 — Ablation sweep (the subtraction half)

Goal: Delete rules and skills whose removal does not hurt the eval.

Run the subtraction half of the ratchet (cooper.simson's purge; Cherny's simple mode). Drop each CLAUDE.md rule and each skill ONE AT A TIME, re-run the eval, and record the score without it. Delete anything whose removal does not drop the score beyond the step-2 noise floor - EXCEPT safety/money/security rules, which are never deleted regardless of eval. Produce an ablation table: item, score with, score without, delta, deleted/kept/locked. The goal is the smallest rule set that holds the score; stale rules written for older models are the primary target.

Expect: An ablation table over every rule and skill, with deletions made for items whose removal did not hurt beyond noise, safety rules marked locked.

Audit before moving on: Confirm each deletion was decided by a real re-run and the delta is within noise. Confirm no safety/money rule was deleted. Fail if nothing was deleted (no real ablation) or a guardrail was cut. On fail, run the ablations properly and restore any guardrail.

Step 4 — SkillOpt-style gated optimizer

Goal: Add edits only when they beat the benchmark; reject the rest.

Build the addition half as a SkillOpt-style loop (github.com/microsoft/SkillOpt): (1) skills log their own failure notes as they run (angus.sewell's issues-table pattern). (2) An optimizer reads the notes and the eval failures and proposes an edit to a rule or skill. (3) The edit is applied to a copy, the eval is re-run, and the edit MERGES only if it beats the current benchmark beyond the noise floor; otherwise it is auto-rejected (SkillOpt's core behaviour). Isolate successful paths from failures. Put the loop on a nightly cron. Log every proposed edit with its before/after score and the merge/reject decision. Never let the optimizer touch the locked safety set.

Expect: A gated optimizer loop (log notes -> propose -> eval -> merge only if it beats baseline beyond noise) on a nightly cron, with an edit log and the safety set excluded.

Audit before moving on: Inspect the edit log: confirm every MERGED edit beat baseline beyond noise and every rejected one did not, and that no edit touched the locked set. Fail if a non-improving edit merged or a safety rule was edited. On fail, fix the gate threshold and the exclusion.

Step 5 — Bilevel outer loop + lock safety

Goal: Revise the optimizer's strategy, and make risk rules uneditable.

Add the two robustness pieces. (1) Bilevel outer loop (raycfu): wrap the SkillOpt loop with an outer loop that periodically revises the OPTIMIZER'S strategy - what kinds of edits it proposes - not just the skill text, so it does not keep trying the same few instincts (raycfu: single loops collapse onto a few favourite ideas). Measure whether the bilevel version explores more distinct edit types than the single loop. (2) Lock the safety set: move all safety/money/security/risk rules into a file or section the loop reads but cannot modify, enforced in code (raycfu's hard-coded-risk-limits pattern). Confirm the optimizer cannot write to it.

Expect: A bilevel outer loop shown to diversify edit types, and a code-enforced lock on the safety/risk rule set.

Audit before moving on: Compare edit-type diversity with and without the outer loop; confirm the optimizer physically cannot modify the locked file. Fail if the outer loop just re-runs the same edits or the safety file is writable by the loop. On fail, fix the strategy-revision step or the lock.

Step 6 — Validate, export traces, gap report

Goal: Prove the ratchet beats baseline beyond noise; export winners; report honestly.

Final validation. (1) Run the final ratcheted CLAUDE.md+skills against the held-out eval vs the starting set; report the gain and confirm it exceeds the step-2 noise floor. Apply wassimyounes_'s caveat: a sub-noise margin is NOT an improvement - say so if that is the case. (2) Export the passing traces (the task runs the winning rule set got right) in a format suitable for a Soup (martintrainhard) small-model fine-tune, and note whether a fine-tune is worthwhile. (3) Enumerate claims, mark verified/unverified/false, list missing pieces ranked by impact, fix the top two. Commit the eval, harness, and ratcheted setup; print the hash.

Expect: A final-vs-baseline eval comparison with a gain exceeding noise (or an honest null result), an exported passing-trace set for Soup, a claim ledger, and a committed setup with hash.

Audit before moving on: Confirm the final gain exceeds the noise floor; if not, state plainly the ratchet did not beat baseline. Confirm the exported traces are the passing ones, not all runs. Fail if a within-noise margin is reported as a win. On fail, report the null result honestly.

4 · Gap analysis and audit

Run this after the build. It makes the agent enumerate its own claims, mark what is verified, rank the gaps and fix the top ones.

Run after the ratchet is built. Enumerate every claim as a numbered list: the eval's objectivity and representativeness, the noise floor, each ablation deletion and keep, each merged/rejected optimizer edit, the bilevel loop's added diversity, the safety lock, and the final gain. Mark each VERIFIED (with run numbers), UNVERIFIED, or FALSE. Specifically re-check: were the eval criteria objective and the set truly held out; is the noise floor below the gains claimed (the falsifier); did the ablation actually DELETE rules, or only keep; did every merged edit beat baseline beyond noise and every rejected one fail; is any final gain within the noise band (a 0.1%-style non-result); is the safety/risk set genuinely uneditable by the loop; are the exported traces the passing ones. List missing pieces: subjective criteria, within-noise gains kept, no deletions, a writable safety file. Rank by impact on whether the ratchet is a real, safe improvement. Fix the top three and re-verify. Report what you could not fix and why. Do not report any improvement that is within the noise floor as a win.

Questions the agent must ask itself before it says done:

5 · What proves it right

6 · Failure modes

7 · Agent direction (unattended runs)

Reach first for the exported chat history and build the held-out eval, then measure the noise floor - the dive's falsifier gates everything, so do not build the ratchet until the eval is proven to carry signal. Cache the eval, the noise floor and the baseline score; ablations and optimizer edits all re-run against them. Run the subtraction half (ablation) and the addition half (SkillOpt-style gated optimizer) as separate, logged passes; spawn a sub-agent for the nightly optimizer loop so proposed edits are evaluated in isolation before merging. Never let any loop edit the safety/money/risk set - enforce that in code, not instruction. Apply the Prime Agent caveat: a sub-noise margin is not an improvement. Stop and ask the user only if there is no exported history to build an eval from. Return a markdown report (eval description, noise/falsifier verdict, ablation table, gated-edit log, locked safety set, 'Unverified and missing') plus the committed eval, harness and ratcheted CLAUDE.md/skills; print the commit hash, and export passing traces for a possible Soup fine-tune. If the ratchet does not beat baseline beyond noise, report that null result honestly.

Prompt system updated 2026-10-22 · all prompt systems · JSON

Browse more deep dives at DejaViewed — 846 entries across the full catalog.

Generated 2026-10-22 · curated by @6ab3 for DejaViewed