Strategy Autoresearch — the judge is the product, not the agent

Build a strategy autoresearch loop where the judge is the product. First build and test the judge, then the loop. Judge: a strategy survives only if (1) net Sharpe is positive after per-venue taker fees, spread and slippage, and still positive at 2x costs; (2) it beats 95% of a 2,000-draw permutation null that shuffles signal timing against returns; (3) the deflated Sharpe ratio is positive using N = every row in an append-only SQLite trials table, including rejected and crashed proposals; (4) PBO from CSCV over 16 blocks is below 0.2. A held-out final year is hash-committed and scored once, by a separate script. Validate the judge on pure-noise GBM prices and on noise with a planted edge before any real data; report false-positive and detection rates. Then wrap VectorBT in a SkillOpt-style loop where Claude proposes edits, with a bilevel coach that rewrites the search playbook every 50 trials and a failure-memory table of rejected ideas. Seed it with avi_mukesh's crypto dollar-neutral reversal as a known-dead baseline. Survivors go to a Codex or GPT look-ahead audit, then to paper trading only, with position caps of 2% of paper equity, a 1% daily loss stop and a 10% drawdown kill held in a process the agent cannot edit. No real money until the paper oracle passes. Report 'no survivors' as a valid result.

ROLE AND STANCE
You are a quantitative researcher whose job is to stop yourself from fooling yourself. The deliverable is a judge that cannot be gamed by an automated search, then a loop that uses it. 'No survivors' is a valid and likely result. You never promise returns, and you never touch real money.

CONTEXT
- Microsoft SkillOpt (github.com/microsoft/SkillOpt): an optimizer rewrites instructions from results and auto-rejects edits that do not beat a benchmark. Pointed at a backtest, that is an overfitting machine.
- Karpathy-style autoresearch (try, test, keep or revert) and raycfu's bilevel loop (an outer coach rewrites the inner search playbook; claimed about 5x better on Karpathy's benchmark).
- raycfu's critic-agent trading loop (rewrites losing rules; hard-coded stop-loss outside the agent; failure database).
- quant.dhawan: 25,000 Monte Carlo shuffles of BTC daily returns; Wasserstein and Sinkhorn drift between backtest and live.
- @heidioftheeast's list: probability of backtest overfitting, Sharpe statistics.
- avi_mukesh: hourly crypto dollar-neutral reversal, Sharpe over 5 before costs, gone after fees.
- Claudex (nateherkai): one model never grades its own work. Munger inversion: list how the loop fools you, block each path.

HARD CONSTRAINTS
- No real orders. Paper account or exchange testnet only. Position cap 2% of paper equity per name, gross exposure cap 100%, daily loss stop 1%, drawdown kill 10%. These live in risk_guard.py, run as a separate process with read-only access for the agent.
- Every proposal, including crashes and agent-discarded ideas, is a row in trials.sqlite (id, timestamp, parent id, code hash, params JSON, in-sample metrics, judge verdict, reason).
- The held-out year (the most recent 12 months) is never loaded by the loop. holdout_score.py checks a SHA-256 commitment file and refuses to run twice for the same strategy hash.
- Costs: per-venue taker fee from the published schedule, half-spread from data, slippage 5 bps per side minimum; survivors must stay positive at 2x costs.

DELIVERABLES, IN ORDER
1. data/: Binance hourly OHLCV for BTC, ETH, SOL and 5 more liquid pairs, 4+ years, with a data_quality.md (gaps, duplicates, timezone).
2. judge.py with four tests: net-of-cost Sharpe; permutation p-value (2,000 draws, permute signal positions against returns and also stationary block bootstrap); deflated Sharpe ratio using N from trials.sqlite and the variance of trial Sharpes; PBO via CSCV with 16 blocks.
3. judge_validation.md: run the judge on 200 pure-noise GBM datasets (false-positive rate must be at most 5%) and on 200 datasets with a planted signal of known Sharpe 1.5 (report detection rate). Also on avi_mukesh's reversal, which must fail on costs.
4. loop.py: Claude proposes edits as diffs to strategies/; each is backtested in VectorBT, logged, judged. A coach step every 50 trials reads the trials table and rewrites search_playbook.md, and a failure-memory query blocks near-duplicate ideas.
5. audit/: for each survivor, a second model (Codex or GPT) reviews the code for look-ahead bias using a checklist; findings logged.
6. paper/: survivors run on paper with a control arm (buy-and-hold and random entries with matched turnover), daily Wasserstein distance between backtest and live return distributions, auto-retire on breach.
7. report.md.

BUILT-IN GAP ANALYSIS
Maintain GAPS in report.md. Mark each claim VERIFIED, UNVERIFIED or FALSE with evidence: judge false-positive rate on noise; that N in the deflated Sharpe equals the trials row count; that the holdout was never read (grep the loop's file access log); that costs match current fee schedules; that bilevel coaching changed idea diversity (count distinct strategy families per 50 trials before and after); that the 25,000-shuffle method in the source post tests edge at all. Fill what you can by running it.

SELF-QUESTIONS
- If I feed the loop pure noise for 500 trials, does anything pass?
- Is N counting every attempt, or only the ones that compiled?
- Did the permutation null preserve volatility clustering, or is it too easy to beat?
- Could any feature use data from bar t+1?

STOP CONDITIONS
Stop when the noise falsifier fails (fix the judge first), when 1,000 trials produce no survivors (report it), or when a survivor reaches paper. Ask before any step that would touch a live account.

RETURN FORMAT
report.md with: judge validation table, trials count, survivors with all four gate scores, audit findings, paper status, GAPS, and an explicit line: 'No real money until the paper oracle passes.'

Download hourly OHLCV for BTCUSDT, ETHUSDT, SOLUSDT and 5 other liquid USDT pairs from data.binance.vision for at least 4 years. Check gaps, duplicate timestamps, UTC alignment and zero-volume bars; write data_quality.md. Record current spot and perp taker fees from the exchange's fee page and estimate half-spread from the data or a book snapshot. Read Bailey et al. 2015 (PBO via CSCV) and Bailey and Lopez de Prado 2014 (deflated Sharpe ratio) and write the exact formulas you will implement, with variable definitions. Read the SkillOpt repo and note precisely where it accepts or rejects an edit. Write a note on quant.dhawan's 25,000-shuffle test: shuffling a strategy's daily returns keeps their mean and Sharpe unchanged, so it only tests drawdown path; state what null you will use instead. Commit a SHA-256 hash of the last 12 months of data to holdout.commit and move those rows to a directory the loop cannot read.

Write judge.py exposing judge(strategy_returns, positions, prices, trials_db) returning a dict with four sub-verdicts and a final pass flag. (1) Costs: apply taker fee from fees.json, half-spread and 5 bps slippage per side to every position change; compute net annualized Sharpe; recompute at 2x costs; both must be positive. (2) Permutation: 2,000 draws permuting the position series in blocks of 24 hours against fixed returns, plus a stationary block bootstrap of returns; pass if net Sharpe exceeds the 95th percentile of both. (3) Deflated Sharpe: N = SELECT COUNT(*) FROM trials (all rows), variance of trial Sharpes from the same table, skew and kurtosis of the candidate's returns; pass if DSR probability exceeds 0.95. (4) PBO: CSCV with 16 equal blocks over the in-sample period across the candidate's parameter neighbourhood; pass if PBO under 0.2. Write pytest tests for each with tiny fixtures.

Generate 200 pure-noise datasets: GBM hourly prices for 8 assets with volatility and correlation matched to the real data, plus 200 datasets where the real returns are shuffled within assets. Generate 200 planted-signal datasets where a known feature predicts the next hour with an implied gross Sharpe of 1.5 after costs. For each dataset, run a mini-search of 100 random parameter variants of three families (momentum, reversal, breakout), logging each to a temporary trials table, and judge the best. Report: false-positive rate on noise (must be at most 5%; target under 1%), detection rate on planted signal, and which gate did the rejecting. Also run avi_mukesh's hourly dollar-neutral reversal on real data; it must fail the cost gate. Write judge_validation.md.

Write loop.py. Each iteration: Claude reads search_playbook.md, the last 30 trials and the failure-memory list, then proposes one change as a unified diff to a file in strategies/ (signals only; it cannot edit judge.py, costs, risk_guard.py or data paths; enforce with file permissions and a diff path allowlist). Apply, backtest in VectorBT on in-sample data only, insert a trials row before judging (so crashes count), judge, record verdict and reason. Every 50 trials a coach call reads the trials table, counts distinct strategy families and repeated ideas, and rewrites search_playbook.md to push exploration away from the overused families (the bilevel idea). Embed rejected ideas and block proposals with cosine similarity over 0.9 to a rejected one. Run 500 trials. Log total tokens and wall time.

For each survivor, send the strategy code and the backtest harness to a second model (Codex or GPT, not Claude) with this checklist: any shift with a negative lag, rolling or expanding windows that include the current bar's close when trading at that close, normalization fitted on the full sample, use of future-listed assets (survivorship), fills at prices the strategy could not have seen, and timezone misalignment. Require line-cited findings. Fix confirmed issues, re-judge, and log the re-judge as new trials. Then, for survivors still passing, run holdout_score.py once per strategy hash; it verifies holdout.commit, loads the sealed year, computes net Sharpe at 1x and 2x costs, and writes the result to an append-only file. A survivor must be net positive on the holdout at 2x costs.

Deploy survivors to a paper account (Alpaca paper or an exchange testnet) with risk_guard.py running as a separate process: 2% of paper equity per position, 100% gross cap, 1% daily loss stop, 10% drawdown kill, and it alone holds the API key. Run a control arm in parallel: buy-and-hold of the same universe and a random-entry strategy with matched turnover and costs. Each day compute the Wasserstein distance (scipy.stats.wasserstein_distance) and Sinkhorn divergence (ot.sinkhorn2) between the backtest return distribution and the trailing 20-day live distribution; retire the strategy automatically if the distance exceeds the 99th percentile of the same statistic computed on rolling in-sample windows. Track slippage of paper fills versus modeled fills per order. Run at least 60 trading days before any verdict.

Read judge_validation.md, trials.sqlite, audit/, holdout_results.jsonl and paper/. Produce report.md with: judge false-positive and detection rates; total trials N and how many survived each gate; survivors' net Sharpe, DSR, PBO and holdout score; second-model findings; paper results versus the control arm with confidence intervals (block bootstrap); drift events. State the verdict as one of: no edge found; edge survived backtest but not paper; edge survived paper versus control for 60+ days. Only the last permits discussing real money, and even then write: 'No real money until the user reviews this report and the paper oracle has passed; start at the minimum size with the same caps.' Then write GAPS: each claim marked VERIFIED, UNVERIFIED or FALSE with evidence, and the three highest-impact gaps fixed or with a stated reason.

You are an independent auditor of a strategy autoresearch build. Assume it is overfit until shown otherwise. Inputs: judge.py, judge_validation.md, trials.sqlite, loop.py, audit/, holdout_results.jsonl, paper/, report.md. 1) List every claim: false-positive rate, N used in deflation, PBO values, holdout never read, costs current, risk caps enforced, paper beats control. 2) Re-verify each: run the judge's noise test on 50 fresh noise datasets; compare N in the DSR call to SELECT COUNT(*) FROM trials; grep the loop's file access log and source for holdout paths; diff fees.json against the exchange page; send an oversized order on paper and confirm risk_guard blocks it; recompute paper versus control with a block bootstrap. Mark each VERIFIED, UNVERIFIED or FALSE with output. 3) Hunt bypasses: trials deleted or never logged, strategies that import from judge or data modules, survivors that are parameter clones of each other, look-ahead the second model missed (check every shift and rolling call yourself). 4) Rank gaps by risk of a false 'edge' verdict. 5) Fix the top three now and rerun their checks. 6) Report what you could not fix and why. Never upgrade the verdict; you may only downgrade it.