Selling to Agents — metered data for buyers that don't have eyes

I want to sell a niche dataset to AI agents rather than humans. Act as a data-licensing product lead. Start from raw sources: the x402 spec and reference code in coinbase/x402, RFC 9110 on HTTP 402, llmstxt.org, the MCP specification and registry, and the crawler documentation for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Then: (1) audit my DejaViewed catalog JSON field by field and split what I authored (tiers, classifications, syntheses, connection maps) from what I do not own (creators' captions and transcripts); only the first may be sold; (2) design a citation-ready record schema with stable IDs, provenance and a citation string; (3) ship a free llms.txt plus a summary endpoint, and a richer x402-metered endpoint tested end to end on Base Sepolia; (4) wrap the same data as an MCP server and list it; (5) log every request with verified bot identity; (6) run a 60-day A/B with arms free, 0.1 cent and 1 cent per call, with success thresholds written before launch. Prove it with on-chain settlements from wallets I do not control and a weekly citation probe. Do not use self-assertion SEO tricks, and do not promise revenue.

ROLE AND STANCE
You are a data-licensing product lead and backend engineer. Your buyer is an agent that never sees a landing page. You compete on structure, provenance and price, never on manipulation. You treat demand as unproven until a wallet you do not control pays you.

CONTEXT
- x402 revives HTTP 402 Payment Required so an agent pays a stablecoin micropayment per request (alexanderelorenzo's post frames it as a toll booth for scrapers; the framing is hype-heavy, the protocol is real).
- Quality publishers block AI crawlers: wetclaude reports Consumer Reports disallows GPTBot, Google's AI crawler and Common Crawl, about 60% of high-credibility news sites block AI crawlers, and one in six cited sources is AI-written. Clean, permitted data is scarce.
- The Princeton study: 18 of 23 assistants favored a sponsored option, GPT-5.1 surfaced it 94% of the time, and a 'serve the customer only' system prompt barely moved it. Its own caveat: it measured a hypothetical deployment, not live paid placement.
- Headless software and MCP (Chamath on All-In): agents pick tools on speed, cost and accuracy.
- llms.txt got brycenwood.ai's wrap shop recommended by ChatGPT, built in 5 minutes with Claude Code.
- edward.builds shows Google AI Overviews citing a self-asserted LinkedIn claim the same day. That is the low-quality competition, not a tactic to copy.
- only-cli/oc and Agent Reach show agents want compact, permitted access; do not build on block-evasion.
- The first product is the DejaViewed catalog's api/ JSON.

HARD CONSTRAINTS
- Sell only fields you authored. Creators' captions and transcripts are excluded from paid records; link to the source post instead.
- Testnet (Base Sepolia USDC) until the end-to-end payment test passes 20 times in a row. Production hot wallet holds no more than 50 USD; sweep to cold storage weekly.
- Never log or print private keys; load them from environment at runtime only.
- Respect robots.txt and platform terms for any data you gather.
- No promised revenue anywhere in docs or listings.

DELIVERABLES, IN ORDER
1. rights_audit.md: every field in the catalog JSON, its origin, sellable yes/no.
2. schema/record.schema.json: id, title, tier, class, summary_authored, connections, source_url, updated_at, citation string, licence.
3. llms.txt and llms-full.txt at the site root describing the dataset and pointing to the free and paid endpoints.
4. api/: GET /v1/free/{id} (title, tier, one-line summary, source link) and GET /v1/records?query= behind x402 middleware, with JSON Schema validation in tests.
5. pay_test.md: 20 end-to-end payments on Base Sepolia using the x402 client from the same repo.
6. mcp/: an MCP server exposing search_records and get_record, with paid calls passing through the same x402 gate, and a registry listing.
7. logging: per request timestamp, path, arm, user agent, verified_bot (reverse DNS forward-confirmed), paid, amount, tx hash.
8. experiment.md: arms free / 0.1 cent / 1 cent assigned by hash bucket, 60-day duration, success thresholds written before launch.
9. probe/: a weekly script that asks Perplexity and ChatGPT 30 fixed questions your data answers, records citations, and checks the cited page contains the claim.

GAP ANALYSIS (BUILT IN)
Write gaps.md as you go: which x402 middleware version you used and whether it supports your host; whether any major agent framework ships an x402 client by default; whether crawlers you see can pay at all; which records lack provenance; which bot identities could not be verified. Close what you can by testing; list the rest with the missing input.

SELF-QUESTIONS
- If I remove every field I do not own, is the remaining data still worth paying for?
- Would an agent pick my record over a free AI-written listicle, and on what visible signal?
- Can a crawler that hits 402 do anything except leave?
- Am I counting spoofed user agents as agent demand?
- Is the free arm cannibalizing the paid one, and does that matter for the falsifier?

STOP CONDITIONS
Stop and ask if the rights audit leaves fewer than 30% of fields sellable, or if testnet payments fail after two fixes. Otherwise stop when all nine deliverables exist and the experiment is live.

RETURN FORMAT
Table of deliverable, path, status; the rights audit summary; pay_test result; the pre-registered thresholds; top five gaps.

Do not write server code yet.
1. Load the DejaViewed catalog JSON from the site's api/ output. For every field, record example value, origin (authored by me, derived by my pipeline from my own judgment, or copied from a creator's post), and sellable yes/no. Captions, transcripts, creator handles' bios and thumbnails are not yours; tiers, classes, deep-dive theses, connection maps and your summaries usually are. Write rights_audit.md and the percentage of sellable fields.
2. Clone coinbase/x402. Read the spec document and list: the 402 response body fields, the payment header the client sends, the facilitator's verify and settle steps, supported networks and tokens, and the server middleware packages available for your stack. Quote file paths.
3. Read RFC 9110 section on 402, the llms.txt proposal at llmstxt.org, and the MCP specification's tool and transport sections.
4. Collect official docs and IP verification methods for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended.
5. Write claims.md testing the dive's inputs: '18 of 23 models favored the sponsored option', '60% of high-credibility news sites block AI crawlers', 'one in six cited sources is AI-written', 'x402 was coded into the internet in 1991 and remained dormant until 2025'. Mark which you can check from primary sources.

Write schema/record.schema.json (JSON Schema 2020-12) for one record: id (stable, never reused), title, tier, class, summary_authored, connections (array of ids with a one-line reason), source_url, updated_at (ISO 8601), licence, and citation (a ready string like 'DejaViewed catalog, record <id>, updated <date>, <url>'). Add a records/ export script that emits only sellable fields from rights_audit.md and validates every record with jsonschema.
Write llms.txt following llmstxt.org: H1 name, a blockquote summary, then sections listing the free endpoint, the paid endpoint with its price, the MCP server, and the licence. Write llms-full.txt with the schema and 5 example records.
Design the response format for agents, borrowing from the only-cli/oc idea: compact, numbered, no HTML, with a total token estimate per response. Measure tokens for a typical response with a tokenizer and keep the free summary under 300 tokens.

Implement api/ with the x402 server middleware from coinbase/x402 for your stack (Hono or Express on Cloudflare Workers or Node).
- GET /v1/free/{id}: title, tier, one-line summary, source link. No payment.
- GET /v1/records?query=&limit=: full sellable records, ranked by a simple BM25 over titles and summaries. Protected by x402 with a price set from an env variable.
- The receiving address comes from env. Never print or log keys; check existence only.
Configure Base Sepolia USDC. Write a client script using the x402 client from the same repo that requests /v1/records, receives 402, pays, retries and gets 200. Run it 20 times. Record each tx hash and latency in pay_test.md.
Then write tests: unpaid request returns 402 with a valid body; paid request returns 200 with schema-valid records; a replayed payment header is rejected.

Build mcp/ with the official MCP TypeScript SDK or the Python mcp package (FastMCP). Expose two tools: search_records(query, limit) and get_record(id). Write tool descriptions an agent will choose on: state the dataset scope, freshness, price per call and that records carry provenance. The server calls your own HTTP API; paid calls go through the x402 client with a per-session spend cap set by the user, defaulting to 10 cents.
Test locally with the MCP Inspector and with Claude Code as a client: install it, ask a question the data answers, confirm the tool is chosen and the answer cites a record id.
Publish to the official MCP registry following its current submission process, and add the server to llms.txt. Log every MCP call through the same request log as the HTTP API, tagged source=mcp.

Add structured logging to every endpoint: timestamp, path, arm, user_agent, client_ip_hash, verified_bot, paid, amount, tx_hash, latency_ms. Store in D1 or R2 on Cloudflare, or SQLite on a VPS.
verified_bot: for user agents claiming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot or Google-Extended, check the IP against published ranges or do reverse DNS plus forward-confirm. Unverified claims are labelled spoofed.
Assign arms by a stable hash of client fingerprint into free, 0.1 cent and 1 cent, so a repeat client stays in one arm. The free arm still serves full records; it is the control.
Write a daily report script: requests by arm and verified bot, paid calls, revenue, unique payer wallets, 402 bounce rate (402 served with no retry). Hash client IPs with a salted SHA-256; never store raw IPs beyond verification.

Write experiment.md before launch: hypothesis, arms, duration 60 days, primary metric (paid calls from distinct wallets you do not control), secondary metrics (402 bounce rate, verified agent requests, citation rate). Pre-register thresholds: for example 'success if at least 25 paid calls from at least 5 distinct external wallets in either paid arm; failure if verified agent requests exceed 1,000 and paid calls are under 5'. Do not change thresholds after launch.
Build probe/: 30 fixed questions the catalog answers well (for example 'best open-source tool for X per DejaViewed'). Weekly, query Perplexity and ChatGPT with browsing, record the answer and every cited URL, and check whether the cited page actually contains the claimed answer, following wetclaude's advice to click every citation. Store results by week.
Do not seed self-assertion posts to game AI Overviews; the probe measures organic pickup only.

At day 60, compute: paid calls and revenue per arm, distinct external payer wallets (exclude every wallet you funded), verified agent requests per arm, 402 bounce rate, and citation rate trend from probe/. Verify a sample of 10 settlements on-chain.
Apply the falsifier: plenty of free crawls but close to zero paid calls means the demand side does not exist yet; write the verdict as 'agents buying', 'agents crawling, not paying', or 'no agent traffic'.
Write gaps.md: every claim from claims.md marked verified / unverified / false with evidence; which agent frameworks could pay x402 during the window; how many 402 responses went to clients that cannot pay; fields you could not sell; anything in logs you cannot explain. Fix what you can now and list the rest with owners.

You are auditing an agent-facing data product after a 60-day price experiment. Read rights_audit.md, schema/, llms.txt, api/, pay_test.md, mcp/, the request logs, experiment.md, probe/ results and results.md.
1. Enumerate every claim: fields sellable, payments working, bot traffic volumes, paid calls, revenue, citation pickup, registry listing.
2. Mark each verified, unverified or false with evidence: a log query, a Base explorer tx, a test run.
3. Re-check specifically: no creator caption or transcript text in paid records; thresholds in experiment.md predate launch; payer wallets exclude ones the builder funded; verified_bot uses reverse DNS or published IP ranges; no private key appears in code, logs or history; the free arm served the same records as paid.
4. List missing pieces: terms of service for buyers, a refund path, rate limiting, uptime monitoring, a data freshness policy.
5. Rank gaps by whether they could flip the verdict.
6. Fix the top three now.
7. Report what you could not fix and why.