Benchmark Your Brain — the skill-optimizer loop, pointed at the human

Insight class: Workflow Multiplier · Difficulty: Easy · Quality: 4/5 Connected entries: 9 · Tools: Anthropic, Anthropic Economic Index, ChatGPT, Claude, Database Detective, Grokbot, NotebookLM, O*NET, OpenAI, SkillOpt

In short: The agent world now rewrites skills against benchmarks: SkillOpt rejects any edit that doesn't beat the eval, Karpathy-style autoresearch adds an outer coach loop, and skills log their own failures for batch rewrites. Meanwhile the catalog's learning entries stop at 'turn saves into quizzes'.

Thesis

The agent world now rewrites skills against benchmarks: SkillOpt rejects any edit that doesn't beat the eval, Karpathy-style autoresearch adds an outer coach loop, and skills log their own failures for batch rewrites. Meanwhile the catalog's learning entries stop at 'turn saves into quizzes'. Run the same loop on yourself. Your saves are the training set, auto-generated quizzes are the eval set, and an outer coach rewrites your study playbook whenever your scores plateau.

Why it matters

Most people's save folders are graveyards, and this catalog is one. Closing the loop turns passive saving into measured skill gain, and it reports which topics never stick, so you can drop or re-approach them. The same harness already works for agents, so the build is small.

The evidence — connected entries

Self-study pipeline — saved links to AI-generated study guides and quizzes

Creator: @angus.sewell · Tier: B Tools: NotebookLM

Input loop: saves table → agent groups by topic → study guides + quizzes

Save videos and articles via Chrome extensions or text into a table, then give your agent a skill that groups recent saves by topic and generates study guides and quizzes, optionally through NotebookLM. Active recall instead of passive watching.

View original post

Microsoft SkillOpt — agent skills that rewrite themselves against benchmarks

Creator: @marc.kaz · Tier: S Tools: SkillOpt

Core rule: every playbook rewrite must beat the benchmark or is rejected

Microsoft open-sourced SkillOpt: a base model runs a task, an optimizer rewrites the skill instructions from results, successful paths are separated from failures, and edits that don't beat the benchmark are auto-rejected. Claimed to beat hand-written prompts and TextGrad and to transfer across models.

View original post

Bilevel autoresearch loops — an outer coach loop rewrites the search

Creator: @raycfu · Tier: A Tools: OpenAI

Outer coach loop that stops the inner learner repeating the same failing approach

Builds on Karpathy's autoresearch loop (try change, test, keep or revert). An outer coach loop watches the inner worker and rewrites its search playbook so it stops repeating the same ideas; claimed ~5x better on Karpathy's benchmark with the same models.

View original post

Self-improving skills vs AI teammates — log issues, rewrite skills

Creator: @angus.sewell · Tier: B Tools: Grokbot

Failure logging: each session writes issues for batch rewrites

Argues personified 'AI teammates' are static while skills are adaptable: keep every skill in a database table, have each skill log issues about itself when used into a second table, then periodically run a prompt that reads the accumulated issues and rewrites the skill. Specific workflows with clear end states beat persona prompts.

View original post

Chat-history mining — derive better custom instructions from your AI logs

Creator: @angus.sewell · Tier: B Tools: ChatGPT, Claude

Second data source: mine your AI chat history for where you keep asking the same thing

Export your full ChatGPT or Claude history, have a small model like Jev tag every conversation for mistakes and usage patterns, then feed the analysis to your main assistant to write new custom instructions. A cheaper alternative to building a full personal 'Jarvis' agent.

View original post

Anthropic job index audit — map your tasks to automation vs augmentation

Creator: @angus.sewell · Tier: A Tools: Anthropic, Anthropic Economic Index, O*NET

Curriculum selection: what's worth learning vs automating, at task level

Download Anthropic's job index, combine it with the US Labor Department's O*NET task database for your role, and have an AI split your tasks by Anthropic's automation-versus-augmentation ratio. A second prompt builds a 90-day plan to automate the automatable parts and get excellent at the rest.

View original post

Database Detective — solve crimes by writing SQL queries

Creator: @frostechgmz · Tier: B Tools: Database Detective

Format: learning as a game where the real skill is choosing the right question

Database Detective is a Steam game where you solve absurd police cases by querying tables with SELECT, WHERE and JOIN to rule out suspects. Beginner-friendly with a manual, clear errors and a hint assistant, and it trains asking the right data question first. English only.

View original post

Glenn Gould's humming — thinking music away from the piano

Creator: @dailyhistoryclips · Tier: B

Target state: build an inner model away from the instrument (Gould), not recall trivia

Glenn Gould hummed audibly on nearly every recording despite Columbia engineers' mic tricks. He traced it to practising under a vacuum cleaner's noise, which taught him to hear pieces internally and learn away from the instrument; the voice is the overflow, audible on the Goldberg Variations.

View original post

5 ChatGPT learning prompts — 80/20, intern mode, skill plan

Creator: @yourchatgptguide · Tier: S Tools: ChatGPT

Prompt seeds: Pareto learning, 30-day skill plan

@yourchatgptguide's list of 5 stock prompts: Pareto learning, report-writing intern, 30-day skill plan, summarize-to-bullets, self-prompt generator. Decent template bank for study/research workflows.

View original post

Action sketch — how to implement this

Export the last 60 days of saves (DejaViewed parquet works) into a table; a weekly Claude skill clusters them into 3 topics and writes 15 retrieval questions each, held out from the study notes. Quiz yourself cold, log scores per topic; the coach skill rewrites the study playbook (spacing, format: explain-back, build-a-thing, game) only if next week's held-out score beats the last. Add chat-history mining monthly: questions you re-asked the AI become forced quiz items. Falsifier: if held-out scores after 6 weeks are no better than a no-coach control topic studied with plain flashcards, the outer loop adds overhead not learning.

Prerequisites

• A saves export (this catalog qualifies)
• Claude/agent with scheduled skills
• ~20 min/week of honest self-quizzing

Key tools and resources

• Anthropic
• Anthropic Economic Index
• ChatGPT
• Claude
• Database Detective
• Grokbot
• NotebookLM
• O*NET
• OpenAI
• SkillOpt

Creators to follow

• @angus.sewell
• @dailyhistoryclips
• @frostechgmz
• @marc.kaz
• @raycfu
• @yourchatgptguide

Prompt system — run this yourself

Everything below is pasteable into Claude Code, Claude, or any agent. Use the one-shot when you want the whole result in one run; use the step prompts when you want to steer. Every step carries its own audit, and the gap audit at the end makes the agent find and fill what it missed.

Mission. Produces a six-week self-experiment that runs the SkillOpt-style benchmark-gated loop on a human learner: the last 60 days of saves (a DejaViewed parquet export qualifies) become the study set, held-out retrieval quizzes become the eval set, and a coach skill rewrites the study playbook only when the next held-out score beats the last. A plain-flashcard control topic runs alongside. Done means per-topic learning curves, a delayed retention test, and a falsifier verdict on whether the coach loop beats flashcards, plus a list of topics that never stick.

1 · Deeper, not bigger

Before running anything, make the task deeper rather than larger. Ask: what would a top professional do here that I am not asking for; what is the rawest material; which methods and tools; what, other than the model or me, proves it right. Then ask for exactly that.

• What a top professional would do (that you didn't ask for): A learning scientist would design this as a within-subject experiment: topics matched for difficulty and randomly assigned to coach and control conditions, questions written and frozen before study begins, and a delayed test one to two weeks after the last session, because immediate scores mostly measure short-term memory. They would grade with a rubric written in advance, ideally scored blind by a second model or person, and compute the minimum effect detectable with 15 questions per topic. They would use established mechanisms as baselines, retrieval practice and spacing, not invent new ones, and they would watch for question leakage: if the generator sees the study notes, held-out items stop being held out.
• Rawest material to start from: The user's own saves export (DejaViewed parquet with titles, captions, transcripts and takeaways), the user's exported Claude or ChatGPT history (conversations.json) for re-asked questions, the Microsoft SkillOpt repository (github.com/microsoft/SkillOpt) for the accept-only-if-it-beats-the-eval rule, Karpathy's autoresearch loop and the bilevel outer-loop write-up, the FSRS scheduler (open-spaced-repetition) and Anki for the control condition, and the primary studies: Roediger and Karpicke 2006 on the testing effect and Cepeda et al. 2006 on distributed practice. O*NET task lists and the Anthropic Economic Index if topics are chosen for job relevance.
• Methods and tools: Python and pandas or DuckDB on the parquet; embedding clustering (sentence-transformers plus HDBSCAN or k-means with k=3 to 4) to form topics; Claude to generate questions from source transcripts with a separate context that never sees the study notes; a frozen item bank with item IDs; a rubric-graded short-answer format scored 0 to 2; FSRS via py-fsrs or Anki for the control topic; a coach skill implementing SkillOpt's rule (propose playbook edit, accept only if next held-out score beats the incumbent); an issues table where each study session logs what failed, rewritten in batch weekly, as in the self-improving skills post. Analysis: paired comparison of coach versus control with bootstrap intervals; learning curves per topic; delayed-test retention ratio.
• What proves it right (other than you or the model): A delayed test 14 days after the last session, graded blind by a second grader who does not know the condition. Where possible an external task: a SQL topic checked by solving Database Detective or SQLBolt exercises; a coding topic checked by a build that runs. The control topic studied with FSRS flashcards for the same minutes. A frozen item bank with a hash recorded before week 1.

Rewritten prompt — ask for exactly that:

Act as a learning scientist running a six-week within-subject experiment on me. Load my saves export (DejaViewed parquet) from the last 60 days. Cluster saves into topics with embeddings and pick four topics of similar size and difficulty; randomly assign three to the coach condition and one to a plain-flashcard control, and record the assignment and seed. For each topic, generate 15 held-out retrieval questions plus a 2-point rubric per question in a separate context that sees only the source transcripts, never my study notes; freeze the bank and record its SHA-256. Write the study playbook v1 (spacing, explain-back, build-a-thing, game) and a coach skill that proposes one playbook change per week and accepts it only if my next held-out score on coach topics beats the incumbent, SkillOpt style. The control topic uses FSRS flashcards for the same minutes per week. Each study session logs issues to a table; the coach reads them. Monthly, mine my exported chat history for questions I re-asked and add them as forced items to a separate bank. At week 6 plus 14 days, give a delayed test graded blind by a second grader. Report learning curves per topic, coach minus control with bootstrap intervals, the minimum detectable effect, topics that never stuck, and whether the falsifier fired.

2 · One-shot prompt

ROLE AND STANCE
You are a learning scientist and agent engineer. You will run the benchmark-gated skill-optimization loop on a person instead of a prompt. Source claims to test: SkillOpt rejects any skill edit that does not beat the benchmark and beats hand-written prompts; a bilevel outer 'coach' loop is about 5x better on Karpathy's autoresearch benchmark; skills that log their own failures and get batch-rewritten improve; turning saves into quizzes beats passive watching; mining chat history yields better instructions. Treat each as a hypothesis.

CONTEXT
- Study set: the user's saves from the last 60 days; DejaViewed parquet has titles, captions, transcripts, takeaways.
- Eval set: held-out retrieval questions generated from source transcripts.
- Control: FSRS spaced-repetition flashcards (py-fsrs or Anki).
- Optional input: exported Claude or ChatGPT history (conversations.json).
- Budget: about 20 minutes per topic per week of honest self-quizzing.

HARD CONSTRAINTS
1. Item bank frozen before week 1; record its SHA-256. No question is edited after the user sees it.
2. Questions are generated in a context that sees only source material, never study notes, playbooks or prior answers.
3. The coach changes at most one playbook element per week and the change is kept only if the next held-out score on coach topics beats the incumbent.
4. The control topic gets equal minutes and no coach.
5. Quizzes are taken cold: no notes, no search, no AI. Scores are self-reported; the agent must not invent them.
6. Chat-history mining stays local; redact names, emails and credentials before any analysis.

DELIVERABLES IN ORDER
1. topics.json: clusters, chosen topics, difficulty proxy, random assignment with seed.
2. item_bank.parquet: 15 held-out items per topic with rubric, weekly sampling plan, hash.
3. playbook_v1.md and coach skill (prompt plus rules) with the accept-only-if-better gate.
4. session_log.parquet and issues.parquet schemas, plus the weekly script that samples 5 items per topic, collects scores, and runs the coach.
5. chat_mining.py: local pass that finds questions the user asked more than once.
6. analysis.md at week 6 plus 14 days: learning curves, coach minus control with bootstrap intervals, minimum detectable effect, blind delayed-test results, never-stuck topics, falsifier verdict.

BUILT-IN GAP ANALYSIS
Write GAPS: weeks with missed quizzes; any item that leaked into study material; topics whose difficulty was mismatched; items the second grader scored differently by more than one point; claims you could not test. Fill gaps you can, for example replacing a leaked item from the reserve bank and marking the swap.

SELF-QUESTIONS
- Are the coach and control topics comparable in difficulty and prior knowledge, and how do I know?
- With 15 items and 5 sampled per week, what is the smallest improvement I could detect?
- Could held-out items have leaked through study notes the generator produced?
- Is a week-over-week rise just practice on repeated items rather than learning?
- Did the coach's accepted edits improve the delayed test, or only the next-week quiz?

STOP CONDITIONS
Pause and ask if the user misses two consecutive weeks, if fewer than three topics have enough saves for 15 items, or if the second grader disagrees with self-grading on more than 30% of items.

RETURN FORMAT
Falsifier verdict, learning curve table per topic, coach minus control with interval, never-stuck topics, the file list, GAPS, and self-question answers.

3 · Step-by-step prompts

Step 1 — Ground truth: saves, topics and assignment

Goal: Form comparable topics from the raw saves and assign conditions randomly.

Load the saves export (DejaViewed parquet or equivalent) and filter to the last 60 days. For each save, build text from title, caption, transcript and takeaways; drop items with under 50 words of usable text and report how many. Embed with sentence-transformers (all-MiniLM-L6-v2 is enough), cluster with HDBSCAN or k-means, and name clusters from their top terms. Pick four clusters with at least 8 saves each and similar text volume. Ask the user to rate prior knowledge 1 to 5 per topic and use it as a difficulty proxy. Randomly assign three topics to coach and one to control with a recorded seed; if prior-knowledge ratings differ by more than 1 point between control and the coach mean, redraw and log it. Write topics.json with members, volume, prior rating, condition and seed.

Expect: topics.json with four topics, member save IDs, prior ratings, assignments and seed, plus a count of dropped saves.

Audit before moving on: Fail if the control topic is chosen by preference instead of the seed, or if any topic lacks enough source text for 15 distinct questions. Spot-read 3 saves per topic to confirm the cluster is coherent.

Step 2 — Freeze the held-out item bank

Goal: Create an eval set the study process can never see.

In a fresh context that receives only source transcripts and captions for one topic at a time, generate 20 retrieval questions per topic: mix of recall, explain-why, and apply-to-new-case, each with a model answer and a 0 to 2 rubric (2 complete and correct, 1 partial, 0 wrong or blank). Keep 15 as the bank and 5 as a reserve for replacing leaked items. Check each question is answerable from the sources, not trivia about the creator. Write item_bank.parquet with item ID, topic, type, question, rubric, model answer, and reserve flag, then compute its SHA-256 and store it in bank_hash.txt. Write a weekly sampling plan: 5 items per topic per week, balanced so each item appears at most twice across six weeks, plus the full 15 on the delayed test.

Expect: item_bank.parquet with 80 items, bank_hash.txt and sampling_plan.csv.

Audit before moving on: Have a second model answer every question from the sources alone; fail any item it cannot answer or answers ambiguously, and replace from reserve. Confirm the generation context had no access to notes or playbooks.

Step 3 — Playbook v1, coach skill and logging

Goal: Implement the benchmark-gated coach and the self-logging study sessions.

Write playbook_v1.md for coach topics: spacing schedule (sessions on days 1, 3 and 7 of each week), one active format per session chosen from explain-back aloud, build-a-thing, teach-a-friend summary or a game, and a 20-minute cap. Write the coach skill: it reads last week's quiz scores per item and topic, issues.parquet entries, and the playbook; it proposes exactly one change with a stated hypothesis; the change becomes the candidate playbook; after the next quiz, the candidate is kept only if mean held-out score on coach topics beats the incumbent's, otherwise reverted, mirroring SkillOpt's reject rule. Write the session logger: after each session the user records minutes, format used and one line on what failed; these go to issues.parquet. Write the control routine: FSRS flashcards built from the same source material, same minutes.

Expect: playbook_v1.md, coach_skill.md, logger script, control card deck, and table schemas.

Audit before moving on: Simulate two weeks with made-up scores clearly marked SIMULATED and confirm the coach reverts a change that lowered scores. Delete simulated rows afterward. Fail if the coach can edit more than one element or read the item bank.

Step 4 — Weekly loop and chat-history mining

Goal: Run six weeks of quizzes and coach updates, adding re-asked questions monthly.

Each week: present the planned 5 items per topic cold, one at a time; the user answers without notes; grade with the rubric and show the model answer after the user commits; record scores in session_log.parquet. Then run the coach. Never fill in a score the user did not give. Monthly, run chat_mining.py on the user's exported conversations.json locally: redact names, emails, phone numbers and keys; find user questions that are semantically near-duplicates across different dates (embedding cosine above 0.85); map them to topics; write them to a separate forced_items bank, not the held-out bank, so they do not contaminate the eval. Print a weekly dashboard: score per topic per week, accepted and rejected coach changes, minutes studied. Flag any week a topic score drops by more than two items.

Expect: Six weeks of session_log rows, a coach decision log, forced_items.parquet, and weekly dashboards.

Audit before moving on: Fail any week where a topic's minutes differ from the control by more than 25%, or where an item was asked more often than the sampling plan allows. Verify the bank hash weekly.

Step 5 — Delayed test with blind grading

Goal: Measure retention, not short-term recall.

Fourteen days after the last study session, give the full 15-item bank for every topic in random order, cold. Collect the user's written answers. Grade twice: the user self-grades with the rubric, and a second grader (a separate model context or a person) grades the same answers without being told the condition of each topic. Compute agreement and use the second grader's scores where they differ. Where a topic has an external task, run it: for a SQL topic, time to solve three Database Detective or SQLBolt exercises; for a coding topic, whether a small build from memory runs. Write delayed.parquet and grader agreement statistics. Also give the 5 forced items from chat-history mining as a separate, unscored-for-the-verdict section, so their retention is visible without contaminating the comparison.

Expect: delayed.parquet with two grades per answer, an agreement rate, and external task results.

Audit before moving on: Fail if the second grader saw condition labels or the playbook. If agreement is under 70%, sharpen the rubric wording and regrade only with the second grader, noting it.

Step 6 — Validate against oracles and write the gap report

Goal: Decide whether the coach loop beats flashcards and list what remains unproven.

Write analysis.md. Show learning curves per topic across six weeks. Compute coach minus control on the delayed test using second-grader scores, with a bootstrap 95% interval over items, and state the minimum detectable effect given the item count. Report the retention ratio (delayed score divided by week-6 score) per topic. List never-stuck topics: delayed score below 50% of maximum. Answer the falsifier: are delayed held-out scores on coach topics no better than the flashcard control? Then a GAPS section: each source claim (SkillOpt's benchmark gate beats hand prompts; the bilevel loop is about 5x better; self-logging skills improve; quizzes beat passive review; chat mining personalizes) marked verified, unverified or false for this one-person experiment, plus missed weeks, leaked items, and limits of a single subject.

Expect: analysis.md with curves, interval, retention ratios, never-stuck list, falsifier verdict and GAPS.

Audit before moving on: Recompute the coach minus control interval from delayed.parquet with a fresh script. Fail if conclusions generalize beyond one person or if immediate quiz gains are presented as retention.

4 · Gap analysis and audit

Run this after the build. It makes the agent enumerate its own claims, mark what is verified, rank the gaps and fix the top ones.

You are an independent reviewer of a six-week benchmark-your-brain experiment. Open topics.json, item_bank.parquet, bank_hash.txt, sampling_plan.csv, session_log.parquet, issues.parquet, the coach decision log, delayed.parquet and analysis.md. List every claim in analysis.md and mark each verified (cite the table and computation), unverified (state what data would settle it), or false. Check: the bank hash never changed; no held-out item appears in study notes, playbooks or forced items; the control topic was assigned by the recorded seed and got comparable minutes; each coach change was accepted only after beating the incumbent score; delayed grading used a grader blind to condition; no score was entered by the agent without user input; conclusions are limited to one subject and the stated minimum detectable effect. List missing pieces: missed weeks, topics with too few items, external tasks not run. Rank problems by whether they could reverse the falsifier verdict, fix the top three now (for example regrading with the blind grader or recomputing intervals), and report before and after. End with what only another cohort, a longer delay, or a second learner could answer.

Questions the agent must ask itself before it says done:

• Are coach and control topics matched in difficulty and prior knowledge, or did I pick an easy control?
• Could any held-out question have appeared in study notes or a generated summary?
• With 15 items per topic, is the effect I hope for even detectable?
• Is the weekly score rising because items repeat, not because knowledge grows?
• Did the coach ever accept a change on a noise-level difference of one item?
• Are my self-grades systematically kinder than the blind grader's?
• Which saves were too thin to support real questions, and did they bias topic choice?
• Did chat-history mining surface re-asked questions I actually need, and were they kept out of the eval bank?
• Is the Glenn Gould idea of learning away from the instrument (explain-back without notes) tested by any playbook change?
• What would I drop: which topic never stuck and should be re-approached or abandoned?

5 · What proves it right

• A delayed test 14 days after the last session, graded by a second grader blind to condition.
• The flashcard control topic on FSRS with matched minutes, as the comparison arm.
• bank_hash.txt matching the item bank at every weekly check, proving the eval was frozen.
• External tasks where available: Database Detective or SQLBolt exercises for SQL topics, a working build for coding topics.
• An independent recomputation of the coach-minus-control interval from delayed.parquet.

6 · Failure modes

• Eval leakage: question generator saw the study notes, so quizzes test memorized notes. Tell: near-perfect scores in week 1. Fix: regenerate from sources only and replace leaked items from reserve.
• Noise-chasing coach: accepts edits on one-item differences. Tell: playbook flips back and forth weekly. Fix: require a margin at least the week-to-week standard deviation before accepting.
• Unmatched control: the control topic is easier or already known. Tell: control prior-knowledge rating differs by 2 or more. Fix: redraw with seed before week 1.
• Repetition masquerading as learning: same items each week. Tell: scores rise on repeated items only. Fix: follow the sampling plan and analyze first-exposure items separately.
• Self-grading drift: rubric applied more kindly over time. Tell: blind grader disagreement rises across weeks. Fix: use blind grades for all conclusions.
• Dropout: missed weeks break the loop. Tell: gaps in session_log. Fix: schedule quizzes as a recurring routine and pause the coach on missed weeks.

7 · Agent direction (unattended runs)

Start from the local saves export with pandas or DuckDB; no network needed except embedding model download. Cache embeddings, topics.json and the frozen bank; never regenerate the bank after week 1. Use a separate sub-agent context for question generation and another for blind grading, and never give either the playbook or condition labels. Run the weekly loop as a scheduled routine that prompts the user and waits; do not proceed without real answers and never invent scores. Keep chat-history mining strictly local with redaction before any model call. Stop and ask when the user misses two weeks, when topics lack material, or when grader agreement is under 70%. Return analysis.md as the entry point with topics.json, item_bank.parquet, bank_hash.txt, sampling_plan.csv, playbook versions, coach decision log, session_log.parquet, issues.parquet, forced_items.parquet and delayed.parquet.

Prompt system updated 2026-10-22 · all prompt systems · JSON

Related deep dives

Browse more deep dives at DejaViewed — 846 entries across the full catalog.

Generated 2026-10-22 · curated by @6ab3 for DejaViewed