Source confidence: high
See also: Deeper Dive — Strategy Autoresearch — the judge is the product, not the agent →

Strategy Autoresearch — the judge is the product, not the agent

Insight class: Workflow Multiplier · Difficulty: Medium · Quality: 5/5 Connected entries: 11 · Tools: Claude, Claudex, Codex, OpenAI, Python, SkillOpt

In short: Self-rewriting loops (Karpathy-style autoresearch, bilevel coach loops, Microsoft SkillOpt) keep any edit that beats a benchmark. Point one at trading and it becomes an industrial overfitting machine — the backtest is the benchmark, and the loop will beat it ten thousand times.

Key fact Detail
What it covers agents, finance/trading, quant, video production
Tools Claude, Codex, Python, Claudex, OpenAI
Posts covered 11 posts (2 S-tier, 5 A-tier) · top: Microsoft SkillOpt — agent skills that rewrite… · Wasserstein Distance — measuring the… · Claudex skill — Claude and GPT-6 Astra…
Creators @raycfu, @nateherkai, @quant.dhawan
Sources dated Jan 2026 – Oct 2026
Difficulty Medium
Prompt system Run-it-yourself prompts
Last verified 2026-10-22

Thesis

Self-rewriting loops (Karpathy-style autoresearch, bilevel coach loops, Microsoft SkillOpt) keep any edit that beats a benchmark. Point one at trading and it becomes an industrial overfitting machine — the backtest is the benchmark, and the loop will beat it ten thousand times. The quant saves already hold the fix: probability of backtest overfitting, permutation tests, transaction costs and backtest-to-live drift. Wire those in as the loop's judge, and use the loop's own log as the exact count of trials — the number human quants never know.

Why it matters

A solo builder can run more strategy variants in a weekend than a small fund tests in a quarter. The difference between that being an edge and a casino is whether the gate deflates for every attempt. Most likely you learn you have no edge, cheaply, and that's worth money too.

The evidence — connected entries

Microsoft SkillOpt — agent skills that rewrite themselves against benchmarks

Creator: @marc.kaz · Tier: S Tools: SkillOpt

Benchmark-gated self-rewrite engine — the loop shape, with auto-reject of edits that don't win

Microsoft open-sourced SkillOpt: a base model runs a task, an optimizer rewrites the skill instructions from results, successful paths are separated from failures, and edits that don't beat the benchmark are auto-rejected. Claimed to beat hand-written prompts and TextGrad and to transfer across models.

View original post

Creator: @raycfu · Tier: A Tools: OpenAI

Outer coach loop that stops the search collapsing onto favorite ideas — and also multiplies trial count

Builds on Karpathy's autoresearch loop (try change, test, keep or revert). An outer coach loop watches the inner worker and rewrites its search playbook so it stops repeating the same ideas; claimed ~5x better on Karpathy's benchmark with the same models.

View original post

Self-improving trading agent — critic loop that rewrites losing strategies

Creator: @raycfu · Tier: B

The naive version — a critic agent rewriting losing trades, with hard-coded risk limits outside its reach

Workflow pitch: describe a strategy, an agent codes it and wires the broker API, runs a five-year backtest, and a critic agent dissects losing trades and rewrites entry/exit logic before rerunning. Hard-coded stop-loss and drawdown limits sit outside the agent's reach, and failed strategies go to a local DB the agent learns from. Guide is comment-gated.

View original post

10 non-cliche quant research projects — Bayesian BL, RL factors, PBO

Creator: unknown · Tier: A

Probability of backtest overfitting and Sharpe-ratio statistics — the deflating judge

@heidioftheeast's 10 quant project ideas beyond Black-Scholes: probability of backtest overfitting, Bayesian Black-Litterman, survival-based risk, RL factor portfolios, Sharpe statistics, convex portfolio trading, TS vs XS factors, numeraire-invariant hedging, exotic option model risk, regime-switching factors.

View original post

Monte Carlo — 25,000 shuffled BTC returns to separate edge from luck

Creator: @quant.dhawan · Tier: A Tools: Python

Permutation test with 25K shuffled returns — the 'random beats you' floor

Runs 25,000 Monte Carlo simulations by shuffling daily BTC-USD returns to test whether a strategy's performance is genuine edge or statistical luck. One backtest isn't enough — real quants demand statistical significance.

View original post

Crypto dollar-neutral reversal — naive long/short, eaten by fees

Creator: @avi_mukesh · Tier: B

Reversal strategy that dies once fees are counted — costs belong inside the judge, not after it

@avi_mukesh: simple reversal strategy on crypto (short if up, long if down), dollar-neutral scaling. Looks good without costs, transaction fees kill it. Honest starter example for learning backtesting caveats.

View original post

Wasserstein Distance — measuring the backtest-to-live reality gap

Creator: @quant.dhawan · Tier: S Tools: Python

Wasserstein drift between backtest and live distributions — the kill switch after deployment

Uses Wasserstein Distance and Sinkhorn Divergence to mathematically measure drift between backtest distributions (blue) and live market conditions (orange). Detects distribution shift that kills more strategies than bad fees.

View original post

Claudex skill — Claude and GPT-6 Astra cross-audit plan and build

Creator: @nateherkai · Tier: A Tools: Claude, Claudex, Codex

Cross-model audit — the model that writes the strategy never grades it

A free skill that splits work between two frontier models: Claude writes the plan and GPT-6 Astra audits it until it holds, then roles swap so Astra builds and Claude audits. Runs inside Claude or Codex as a reusable skill.

View original post

Claude multi-agent trading team — research, market, backtest and risk agents

Creator: @raycfu · Tier: C Tools: Claude

Dedicated risk agent that audits for self-deception — the role split as a template

Pitch for an unnamed free repo that spins Claude into a four-agent trading team: a research agent cleans market data, a market agent writes strategy logic, a test agent backtests for win rate and max drawdown, and a risk agent audits for self-deception. The repo is never named; the link is gated behind a comment.

View original post

GPT-6 Astra stock-trading challenge — scheduled Codex tasks in one thread

Creator: @nateherkai · Tier: C Tools: Claude, Codex

A one-month LLM trading 'win' — the noise this loop must refuse to reward

Creator gives GPT-6 Astra $10k to trade, rerunning an earlier Claude experiment that reportedly beat the S&P by 8% in a month. Six scheduled Codex tasks per day read news, open, review and manage positions, all posting into one shared thread for visibility. No results yet; setup doc gated behind a comment.

View original post

Charlie Munger's inversion — solve problems by asking how to cause failure

Creator: @charliemungerhub · Tier: A

Munger's inversion — define how the strategy dies before the loop starts searching

Munger explains inversion with his WWII meteorologist job: ask how you'd kill pilots (icing, running out of fuel), then avoid those. Same move for any problem: list what would hurt it most and steer clear.

View original post

Action sketch — how to implement this

Wrap a vectorized backtester (VectorBT) in a SkillOpt-style loop where Claude proposes strategy edits and every proposal is appended to a trials table with its parameters and result. Replace 'beats benchmark' with a judge that needs all four: positive after realistic fees, beats 95% of a permutation null, deflated Sharpe positive using the trials-table count, and a held-out year scored once at the end. Have a second model (Codex or GPT) audit each surviving strategy's code for look-ahead bias before it reaches paper trading; monitor live drift with Wasserstein distance and auto-retire on breach. falsifier: seed the loop with pure-noise synthetic prices. If anything passes the judge, the judge is broken — fix it before you show it a real market.

Prerequisites

Key tools and resources

Creators to follow

Prompt system — run this yourself

Everything below is pasteable into Claude Code, Claude, or any agent. Use the one-shot when you want the whole result in one run; use the step prompts when you want to steer. Every step carries its own audit, and the gap audit at the end makes the agent find and fill what it missed.

Mission. Running this system builds a strategy autoresearch loop whose judge is designed to reject overfitting: Claude proposes strategy edits to a VectorBT backtester, every proposal is logged to a trials table, and a four-part gate (net of realistic fees, beats 95% of a permutation null, positive deflated Sharpe using the logged trial count, one-shot held-out year) decides survival. It is for a solo builder who wants to know cheaply whether they have any edge. Done means the judge rejects every strategy on pure-noise synthetic prices, survivors pass a second-model look-ahead audit, and any survivor runs only on paper with caps until the paper oracle passes. No real money, and no return is promised.

1 · Deeper, not bigger

Before running anything, make the task deeper rather than larger. Ask: what would a top professional do here that I am not asking for; what is the rawest material; which methods and tools; what, other than the model or me, proves it right. Then ask for exactly that.

Rewritten prompt — ask for exactly that:

Build a strategy autoresearch loop where the judge is the product. First build and test the judge, then the loop. Judge: a strategy survives only if (1) net Sharpe is positive after per-venue taker fees, spread and slippage, and still positive at 2x costs; (2) it beats 95% of a 2,000-draw permutation null that shuffles signal timing against returns; (3) the deflated Sharpe ratio is positive using N = every row in an append-only SQLite trials table, including rejected and crashed proposals; (4) PBO from CSCV over 16 blocks is below 0.2. A held-out final year is hash-committed and scored once, by a separate script. Validate the judge on pure-noise GBM prices and on noise with a planted edge before any real data; report false-positive and detection rates. Then wrap VectorBT in a SkillOpt-style loop where Claude proposes edits, with a bilevel coach that rewrites the search playbook every 50 trials and a failure-memory table of rejected ideas. Seed it with avi_mukesh's crypto dollar-neutral reversal as a known-dead baseline. Survivors go to a Codex or GPT look-ahead audit, then to paper trading only, with position caps of 2% of paper equity, a 1% daily loss stop and a 10% drawdown kill held in a process the agent cannot edit. No real money until the paper oracle passes. Report 'no survivors' as a valid result.

2 · One-shot prompt

ROLE AND STANCE
You are a quantitative researcher whose job is to stop yourself from fooling yourself. The deliverable is a judge that cannot be gamed by an automated search, then a loop that uses it. 'No survivors' is a valid and likely result. You never promise returns, and you never touch real money.

CONTEXT
- Microsoft SkillOpt (github.com/microsoft/SkillOpt): an optimizer rewrites instructions from results and auto-rejects edits that do not beat a benchmark. Pointed at a backtest, that is an overfitting machine.
- Karpathy-style autoresearch (try, test, keep or revert) and raycfu's bilevel loop (an outer coach rewrites the inner search playbook; claimed about 5x better on Karpathy's benchmark).
- raycfu's critic-agent trading loop (rewrites losing rules; hard-coded stop-loss outside the agent; failure database).
- quant.dhawan: 25,000 Monte Carlo shuffles of BTC daily returns; Wasserstein and Sinkhorn drift between backtest and live.
- @heidioftheeast's list: probability of backtest overfitting, Sharpe statistics.
- avi_mukesh: hourly crypto dollar-neutral reversal, Sharpe over 5 before costs, gone after fees.
- Claudex (nateherkai): one model never grades its own work. Munger inversion: list how the loop fools you, block each path.

HARD CONSTRAINTS
- No real orders. Paper account or exchange testnet only. Position cap 2% of paper equity per name, gross exposure cap 100%, daily loss stop 1%, drawdown kill 10%. These live in risk_guard.py, run as a separate process with read-only access for the agent.
- Every proposal, including crashes and agent-discarded ideas, is a row in trials.sqlite (id, timestamp, parent id, code hash, params JSON, in-sample metrics, judge verdict, reason).
- The held-out year (the most recent 12 months) is never loaded by the loop. holdout_score.py checks a SHA-256 commitment file and refuses to run twice for the same strategy hash.
- Costs: per-venue taker fee from the published schedule, half-spread from data, slippage 5 bps per side minimum; survivors must stay positive at 2x costs.

DELIVERABLES, IN ORDER
1. data/: Binance hourly OHLCV for BTC, ETH, SOL and 5 more liquid pairs, 4+ years, with a data_quality.md (gaps, duplicates, timezone).
2. judge.py with four tests: net-of-cost Sharpe; permutation p-value (2,000 draws, permute signal positions against returns and also stationary block bootstrap); deflated Sharpe ratio using N from trials.sqlite and the variance of trial Sharpes; PBO via CSCV with 16 blocks.
3. judge_validation.md: run the judge on 200 pure-noise GBM datasets (false-positive rate must be at most 5%) and on 200 datasets with a planted signal of known Sharpe 1.5 (report detection rate). Also on avi_mukesh's reversal, which must fail on costs.
4. loop.py: Claude proposes edits as diffs to strategies/; each is backtested in VectorBT, logged, judged. A coach step every 50 trials reads the trials table and rewrites search_playbook.md, and a failure-memory query blocks near-duplicate ideas.
5. audit/: for each survivor, a second model (Codex or GPT) reviews the code for look-ahead bias using a checklist; findings logged.
6. paper/: survivors run on paper with a control arm (buy-and-hold and random entries with matched turnover), daily Wasserstein distance between backtest and live return distributions, auto-retire on breach.
7. report.md.

BUILT-IN GAP ANALYSIS
Maintain GAPS in report.md. Mark each claim VERIFIED, UNVERIFIED or FALSE with evidence: judge false-positive rate on noise; that N in the deflated Sharpe equals the trials row count; that the holdout was never read (grep the loop's file access log); that costs match current fee schedules; that bilevel coaching changed idea diversity (count distinct strategy families per 50 trials before and after); that the 25,000-shuffle method in the source post tests edge at all. Fill what you can by running it.

SELF-QUESTIONS
- If I feed the loop pure noise for 500 trials, does anything pass?
- Is N counting every attempt, or only the ones that compiled?
- Did the permutation null preserve volatility clustering, or is it too easy to beat?
- Could any feature use data from bar t+1?

STOP CONDITIONS
Stop when the noise falsifier fails (fix the judge first), when 1,000 trials produce no survivors (report it), or when a survivor reaches paper. Ask before any step that would touch a live account.

RETURN FORMAT
report.md with: judge validation table, trials count, survivors with all four gate scores, audit findings, paper status, GAPS, and an explicit line: 'No real money until the paper oracle passes.'

3 · Step-by-step prompts

Step 1 — Ground truth: data, costs, source methods

Goal: Assemble raw data, real fee schedules and the original papers before writing any loop.

Download hourly OHLCV for BTCUSDT, ETHUSDT, SOLUSDT and 5 other liquid USDT pairs from data.binance.vision for at least 4 years. Check gaps, duplicate timestamps, UTC alignment and zero-volume bars; write data_quality.md. Record current spot and perp taker fees from the exchange's fee page and estimate half-spread from the data or a book snapshot. Read Bailey et al. 2015 (PBO via CSCV) and Bailey and Lopez de Prado 2014 (deflated Sharpe ratio) and write the exact formulas you will implement, with variable definitions. Read the SkillOpt repo and note precisely where it accepts or rejects an edit. Write a note on quant.dhawan's 25,000-shuffle test: shuffling a strategy's daily returns keeps their mean and Sharpe unchanged, so it only tests drawdown path; state what null you will use instead. Commit a SHA-256 hash of the last 12 months of data to holdout.commit and move those rows to a directory the loop cannot read.

Expect: data/ with a quality report, fees.json, formulas.md with PBO and DSR math, a SkillOpt note, holdout.commit.

Audit before moving on: Recompute the holdout hash and confirm it matches. Fail if any formula lacks a citation or if the loop's data path includes the holdout dates. Fix before step 2.

Step 2 — Build the judge

Goal: Implement the four-part gate as tested code.

Write judge.py exposing judge(strategy_returns, positions, prices, trials_db) returning a dict with four sub-verdicts and a final pass flag. (1) Costs: apply taker fee from fees.json, half-spread and 5 bps slippage per side to every position change; compute net annualized Sharpe; recompute at 2x costs; both must be positive. (2) Permutation: 2,000 draws permuting the position series in blocks of 24 hours against fixed returns, plus a stationary block bootstrap of returns; pass if net Sharpe exceeds the 95th percentile of both. (3) Deflated Sharpe: N = SELECT COUNT(*) FROM trials (all rows), variance of trial Sharpes from the same table, skew and kurtosis of the candidate's returns; pass if DSR probability exceeds 0.95. (4) PBO: CSCV with 16 equal blocks over the in-sample period across the candidate's parameter neighbourhood; pass if PBO under 0.2. Write pytest tests for each with tiny fixtures.

Expect: judge.py with four functions, pytest passing, a docstring citing each paper.

Audit before moving on: Test that a strategy with zero turnover pays zero costs, that N changes when a row is inserted, and that permutation p-values are uniform on random positions (KS test p over 0.05). Fail and fix on any mismatch.

Step 3 — Validate the judge on noise and planted signals

Goal: Measure the judge's false-positive and detection rates before showing it a market.

Generate 200 pure-noise datasets: GBM hourly prices for 8 assets with volatility and correlation matched to the real data, plus 200 datasets where the real returns are shuffled within assets. Generate 200 planted-signal datasets where a known feature predicts the next hour with an implied gross Sharpe of 1.5 after costs. For each dataset, run a mini-search of 100 random parameter variants of three families (momentum, reversal, breakout), logging each to a temporary trials table, and judge the best. Report: false-positive rate on noise (must be at most 5%; target under 1%), detection rate on planted signal, and which gate did the rejecting. Also run avi_mukesh's hourly dollar-neutral reversal on real data; it must fail the cost gate. Write judge_validation.md.

Expect: judge_validation.md with a table: dataset type, n, pass rate, failing gate counts.

Audit before moving on: If anything passes on noise above 5%, stop: inspect which gate let it through, tighten it, rerun all 600 datasets. Never proceed to step 4 with a failing falsifier.

Step 4 — The autoresearch loop with bilevel coach

Goal: Let Claude search strategy space under the validated judge.

Write loop.py. Each iteration: Claude reads search_playbook.md, the last 30 trials and the failure-memory list, then proposes one change as a unified diff to a file in strategies/ (signals only; it cannot edit judge.py, costs, risk_guard.py or data paths; enforce with file permissions and a diff path allowlist). Apply, backtest in VectorBT on in-sample data only, insert a trials row before judging (so crashes count), judge, record verdict and reason. Every 50 trials a coach call reads the trials table, counts distinct strategy families and repeated ideas, and rewrites search_playbook.md to push exploration away from the overused families (the bilevel idea). Embed rejected ideas and block proposals with cosine similarity over 0.9 to a rejected one. Run 500 trials. Log total tokens and wall time.

Expect: trials.sqlite with 500+ rows, search_playbook.md history, a diversity chart of families per 50 trials, a list of survivors (possibly empty).

Audit before moving on: Confirm row count equals iterations attempted, that no diff touched a protected path, and that the coach changed family diversity. If survivors exist, check whether they share one family with tweaked parameters; if so, treat them as one trial family and recheck PBO.

Step 5 — Cross-model look-ahead audit and holdout

Goal: Have a different model audit survivors, then score the holdout once.

For each survivor, send the strategy code and the backtest harness to a second model (Codex or GPT, not Claude) with this checklist: any shift with a negative lag, rolling or expanding windows that include the current bar's close when trading at that close, normalization fitted on the full sample, use of future-listed assets (survivorship), fills at prices the strategy could not have seen, and timezone misalignment. Require line-cited findings. Fix confirmed issues, re-judge, and log the re-judge as new trials. Then, for survivors still passing, run holdout_score.py once per strategy hash; it verifies holdout.commit, loads the sealed year, computes net Sharpe at 1x and 2x costs, and writes the result to an append-only file. A survivor must be net positive on the holdout at 2x costs.

Expect: audit/ with per-survivor findings, re-judged trials rows, holdout_results.jsonl with one line per strategy.

Audit before moving on: Confirm holdout_score.py refused a second run for the same hash (test it). Fail the step if any survivor reached holdout without a second-model audit.

Step 6 — Paper trading with a control and drift kill

Goal: Test survivors forward on paper only, against a control.

Deploy survivors to a paper account (Alpaca paper or an exchange testnet) with risk_guard.py running as a separate process: 2% of paper equity per position, 100% gross cap, 1% daily loss stop, 10% drawdown kill, and it alone holds the API key. Run a control arm in parallel: buy-and-hold of the same universe and a random-entry strategy with matched turnover and costs. Each day compute the Wasserstein distance (scipy.stats.wasserstein_distance) and Sinkhorn divergence (ot.sinkhorn2) between the backtest return distribution and the trailing 20-day live distribution; retire the strategy automatically if the distance exceeds the 99th percentile of the same statistic computed on rolling in-sample windows. Track slippage of paper fills versus modeled fills per order. Run at least 60 trading days before any verdict.

Expect: paper/ with daily logs, control comparisons, drift series, fills versus model, and retirement events.

Audit before moving on: Check that risk_guard.py actually blocked an oversized test order (send one deliberately on paper). Fail if the agent process holds any exchange key. No real money at this step under any result.

Step 7 — Validate against oracles and write the gap report

Goal: Decide honestly whether anything has an edge.

Read judge_validation.md, trials.sqlite, audit/, holdout_results.jsonl and paper/. Produce report.md with: judge false-positive and detection rates; total trials N and how many survived each gate; survivors' net Sharpe, DSR, PBO and holdout score; second-model findings; paper results versus the control arm with confidence intervals (block bootstrap); drift events. State the verdict as one of: no edge found; edge survived backtest but not paper; edge survived paper versus control for 60+ days. Only the last permits discussing real money, and even then write: 'No real money until the user reviews this report and the paper oracle has passed; start at the minimum size with the same caps.' Then write GAPS: each claim marked VERIFIED, UNVERIFIED or FALSE with evidence, and the three highest-impact gaps fixed or with a stated reason.

Expect: report.md with verdict, tables, GAPS and the no-real-money line.

Audit before moving on: Cross-check every number in report.md against its source file. Fail if the verdict is stronger than the evidence (for example, 'edge' with fewer than 60 paper days).

4 · Gap analysis and audit

Run this after the build. It makes the agent enumerate its own claims, mark what is verified, rank the gaps and fix the top ones.

You are an independent auditor of a strategy autoresearch build. Assume it is overfit until shown otherwise. Inputs: judge.py, judge_validation.md, trials.sqlite, loop.py, audit/, holdout_results.jsonl, paper/, report.md. 1) List every claim: false-positive rate, N used in deflation, PBO values, holdout never read, costs current, risk caps enforced, paper beats control. 2) Re-verify each: run the judge's noise test on 50 fresh noise datasets; compare N in the DSR call to SELECT COUNT(*) FROM trials; grep the loop's file access log and source for holdout paths; diff fees.json against the exchange page; send an oversized order on paper and confirm risk_guard blocks it; recompute paper versus control with a block bootstrap. Mark each VERIFIED, UNVERIFIED or FALSE with output. 3) Hunt bypasses: trials deleted or never logged, strategies that import from judge or data modules, survivors that are parameter clones of each other, look-ahead the second model missed (check every shift and rolling call yourself). 4) Rank gaps by risk of a false 'edge' verdict. 5) Fix the top three now and rerun their checks. 6) Report what you could not fix and why. Never upgrade the verdict; you may only downgrade it.

Questions the agent must ask itself before it says done:

5 · What proves it right

6 · Failure modes

7 · Agent direction (unattended runs)

Run steps strictly in order; step 3 is a hard gate and nothing proceeds while noise passes the judge. Reach first for Python, vectorbt, SQLite and pytest; cache downloaded Binance files and backtest results keyed by strategy code hash plus data range. Delegate data download and the PBO/DSR code to sub-agents; review both against the papers. The second-model audit must use a different provider (Codex or GPT); do not let Claude grade Claude. Never place a real order, never request or print exchange secrets, and keep API keys only in risk_guard.py's environment loaded at runtime. Stop and ask the user before connecting any paper or testnet account and before any discussion of live capital. Return: judge.py and tests, judge_validation.md, trials.sqlite, loop.py, search_playbook.md history, audit/, holdout_results.jsonl, paper/, report.md with the verdict and the line 'No real money until the paper oracle passes.'

Prompt system updated 2026-10-22 · all prompt systems · JSON

Browse more deep dives at DejaViewed — 846 entries across the full catalog.

Generated 2026-10-22 · curated by @6ab3 for DejaViewed