Benchmark Your Brain — the skill-optimizer loop, pointed at the human

Act as a learning scientist running a six-week within-subject experiment on me. Load my saves export (DejaViewed parquet) from the last 60 days. Cluster saves into topics with embeddings and pick four topics of similar size and difficulty; randomly assign three to the coach condition and one to a plain-flashcard control, and record the assignment and seed. For each topic, generate 15 held-out retrieval questions plus a 2-point rubric per question in a separate context that sees only the source transcripts, never my study notes; freeze the bank and record its SHA-256. Write the study playbook v1 (spacing, explain-back, build-a-thing, game) and a coach skill that proposes one playbook change per week and accepts it only if my next held-out score on coach topics beats the incumbent, SkillOpt style. The control topic uses FSRS flashcards for the same minutes per week. Each study session logs issues to a table; the coach reads them. Monthly, mine my exported chat history for questions I re-asked and add them as forced items to a separate bank. At week 6 plus 14 days, give a delayed test graded blind by a second grader. Report learning curves per topic, coach minus control with bootstrap intervals, the minimum detectable effect, topics that never stuck, and whether the falsifier fired.

ROLE AND STANCE
You are a learning scientist and agent engineer. You will run the benchmark-gated skill-optimization loop on a person instead of a prompt. Source claims to test: SkillOpt rejects any skill edit that does not beat the benchmark and beats hand-written prompts; a bilevel outer 'coach' loop is about 5x better on Karpathy's autoresearch benchmark; skills that log their own failures and get batch-rewritten improve; turning saves into quizzes beats passive watching; mining chat history yields better instructions. Treat each as a hypothesis.

CONTEXT
- Study set: the user's saves from the last 60 days; DejaViewed parquet has titles, captions, transcripts, takeaways.
- Eval set: held-out retrieval questions generated from source transcripts.
- Control: FSRS spaced-repetition flashcards (py-fsrs or Anki).
- Optional input: exported Claude or ChatGPT history (conversations.json).
- Budget: about 20 minutes per topic per week of honest self-quizzing.

HARD CONSTRAINTS
1. Item bank frozen before week 1; record its SHA-256. No question is edited after the user sees it.
2. Questions are generated in a context that sees only source material, never study notes, playbooks or prior answers.
3. The coach changes at most one playbook element per week and the change is kept only if the next held-out score on coach topics beats the incumbent.
4. The control topic gets equal minutes and no coach.
5. Quizzes are taken cold: no notes, no search, no AI. Scores are self-reported; the agent must not invent them.
6. Chat-history mining stays local; redact names, emails and credentials before any analysis.

DELIVERABLES IN ORDER
1. topics.json: clusters, chosen topics, difficulty proxy, random assignment with seed.
2. item_bank.parquet: 15 held-out items per topic with rubric, weekly sampling plan, hash.
3. playbook_v1.md and coach skill (prompt plus rules) with the accept-only-if-better gate.
4. session_log.parquet and issues.parquet schemas, plus the weekly script that samples 5 items per topic, collects scores, and runs the coach.
5. chat_mining.py: local pass that finds questions the user asked more than once.
6. analysis.md at week 6 plus 14 days: learning curves, coach minus control with bootstrap intervals, minimum detectable effect, blind delayed-test results, never-stuck topics, falsifier verdict.

BUILT-IN GAP ANALYSIS
Write GAPS: weeks with missed quizzes; any item that leaked into study material; topics whose difficulty was mismatched; items the second grader scored differently by more than one point; claims you could not test. Fill gaps you can, for example replacing a leaked item from the reserve bank and marking the swap.

SELF-QUESTIONS
- Are the coach and control topics comparable in difficulty and prior knowledge, and how do I know?
- With 15 items and 5 sampled per week, what is the smallest improvement I could detect?
- Could held-out items have leaked through study notes the generator produced?
- Is a week-over-week rise just practice on repeated items rather than learning?
- Did the coach's accepted edits improve the delayed test, or only the next-week quiz?

STOP CONDITIONS
Pause and ask if the user misses two consecutive weeks, if fewer than three topics have enough saves for 15 items, or if the second grader disagrees with self-grading on more than 30% of items.

RETURN FORMAT
Falsifier verdict, learning curve table per topic, coach minus control with interval, never-stuck topics, the file list, GAPS, and self-question answers.

Load the saves export (DejaViewed parquet or equivalent) and filter to the last 60 days. For each save, build text from title, caption, transcript and takeaways; drop items with under 50 words of usable text and report how many. Embed with sentence-transformers (all-MiniLM-L6-v2 is enough), cluster with HDBSCAN or k-means, and name clusters from their top terms. Pick four clusters with at least 8 saves each and similar text volume. Ask the user to rate prior knowledge 1 to 5 per topic and use it as a difficulty proxy. Randomly assign three topics to coach and one to control with a recorded seed; if prior-knowledge ratings differ by more than 1 point between control and the coach mean, redraw and log it. Write topics.json with members, volume, prior rating, condition and seed.

In a fresh context that receives only source transcripts and captions for one topic at a time, generate 20 retrieval questions per topic: mix of recall, explain-why, and apply-to-new-case, each with a model answer and a 0 to 2 rubric (2 complete and correct, 1 partial, 0 wrong or blank). Keep 15 as the bank and 5 as a reserve for replacing leaked items. Check each question is answerable from the sources, not trivia about the creator. Write item_bank.parquet with item ID, topic, type, question, rubric, model answer, and reserve flag, then compute its SHA-256 and store it in bank_hash.txt. Write a weekly sampling plan: 5 items per topic per week, balanced so each item appears at most twice across six weeks, plus the full 15 on the delayed test.

Write playbook_v1.md for coach topics: spacing schedule (sessions on days 1, 3 and 7 of each week), one active format per session chosen from explain-back aloud, build-a-thing, teach-a-friend summary or a game, and a 20-minute cap. Write the coach skill: it reads last week's quiz scores per item and topic, issues.parquet entries, and the playbook; it proposes exactly one change with a stated hypothesis; the change becomes the candidate playbook; after the next quiz, the candidate is kept only if mean held-out score on coach topics beats the incumbent's, otherwise reverted, mirroring SkillOpt's reject rule. Write the session logger: after each session the user records minutes, format used and one line on what failed; these go to issues.parquet. Write the control routine: FSRS flashcards built from the same source material, same minutes.

Each week: present the planned 5 items per topic cold, one at a time; the user answers without notes; grade with the rubric and show the model answer after the user commits; record scores in session_log.parquet. Then run the coach. Never fill in a score the user did not give. Monthly, run chat_mining.py on the user's exported conversations.json locally: redact names, emails, phone numbers and keys; find user questions that are semantically near-duplicates across different dates (embedding cosine above 0.85); map them to topics; write them to a separate forced_items bank, not the held-out bank, so they do not contaminate the eval. Print a weekly dashboard: score per topic per week, accepted and rejected coach changes, minutes studied. Flag any week a topic score drops by more than two items.

Fourteen days after the last study session, give the full 15-item bank for every topic in random order, cold. Collect the user's written answers. Grade twice: the user self-grades with the rubric, and a second grader (a separate model context or a person) grades the same answers without being told the condition of each topic. Compute agreement and use the second grader's scores where they differ. Where a topic has an external task, run it: for a SQL topic, time to solve three Database Detective or SQLBolt exercises; for a coding topic, whether a small build from memory runs. Write delayed.parquet and grader agreement statistics. Also give the 5 forced items from chat-history mining as a separate, unscored-for-the-verdict section, so their retention is visible without contaminating the comparison.

Write analysis.md. Show learning curves per topic across six weeks. Compute coach minus control on the delayed test using second-grader scores, with a bootstrap 95% interval over items, and state the minimum detectable effect given the item count. Report the retention ratio (delayed score divided by week-6 score) per topic. List never-stuck topics: delayed score below 50% of maximum. Answer the falsifier: are delayed held-out scores on coach topics no better than the flashcard control? Then a GAPS section: each source claim (SkillOpt's benchmark gate beats hand prompts; the bilevel loop is about 5x better; self-logging skills improve; quizzes beat passive review; chat mining personalizes) marked verified, unverified or false for this one-person experiment, plus missed weeks, leaked items, and limits of a single subject.

You are an independent reviewer of a six-week benchmark-your-brain experiment. Open topics.json, item_bank.parquet, bank_hash.txt, sampling_plan.csv, session_log.parquet, issues.parquet, the coach decision log, delayed.parquet and analysis.md. List every claim in analysis.md and mark each verified (cite the table and computation), unverified (state what data would settle it), or false. Check: the bank hash never changed; no held-out item appears in study notes, playbooks or forced items; the control topic was assigned by the recorded seed and got comparable minutes; each coach change was accepted only after beating the incumbent score; delayed grading used a grader blind to condition; no score was entered by the agent without user input; conclusions are limited to one subject and the stated minimum detectable effect. List missing pieces: missed weeks, topics with too few items, external tasks not run. Rank problems by whether they could reverse the falsifier verdict, fix the top three now (for example regrading with the blind grader or recomputing intervals), and report before and after. End with what only another cohort, a longer delay, or a second learner could answer.