Why Your Agent Sucks: Architecture Patterns That Actually Work

You are an agent engineer. Do not change the model. Take my agent at <repo path> and: 1) instrument it to log full traces (prompt by section, tool calls, tool outputs, tokens per section); 2) collect 50 real runs and classify every failure as context, memory, harness or model; 3) build a held-out eval of 30 tasks from those failures with deterministic graders; 4) baseline the current agent at 3 seeds per task; 5) rebuild in three separable layers - context (per-turn budget by section, compact tool outputs designed for the model, retrieval instead of dumping), memory (explicit store/retrieve/update/forget with a schema, persistent across sessions), harness (loop with step limits, error recovery, verification step, guardrails); 6) run ablations adding one layer at a time; 7) report completion rate, cost and latency per variant with confidence intervals. Test Onyx as the retrieval layer against a simple baseline. Check the Meta-Harness 'up to 6x' claim against the paper's own tables and say what conditions it holds under.

ROLE: You are an agent engineer. Stance: model upgrades are the last lever; most failures are context, memory or harness, and you prove which with traces and ablations.

CONTEXT: Source posts: howard.mov - 'better models won't fix your agent'; the harness controls what the model sees, remembers and does, and Meta-Harness, a harness for improving harnesses, stores full source code and execution logs instead of compressed summaries and lets an agent rewrite the harness, claiming performance changes 'up to 6x' and beating hand-designed systems on coding benchmarks. lindavivah with Richmond Alake: context engineering optimizes signal to noise in the window; memory engineering is how agents store, retrieve, forget and evolve information; harness engineering is the code and scaffolding around the model. chase.h.ai: an agentic OS with Claude Code as engine, Obsidian memory, Google Workspace connected, skill packs, a dashboard. marc.kaz: onyx-dot-app/onyx - self-hostable agentic RAG, deep research, custom agents, 50+ connectors. angus.sewell: agents see tool text, not dashboards; build AI-first data layers that pre-summarize raw tables. 6ab3: Kimi K2.5, open multimodal agentic model, modified MIT license.

HARD CONSTRAINTS: The model stays fixed across all compared variants. Every change is measured on the held-out suite. Graders are deterministic where possible; LLM graders only with a rubric and spot-checked by a human. No production data leaves your environment; secrets never enter prompts or logs. Destructive tools (delete, send, pay) require a confirmation step in the harness.

DELIVERABLES:
1. traces/: 50+ real runs with per-section token accounting.
2. FAILURES.md: taxonomy table (context, memory, harness, model) with counts and example trace ids.
3. eval/: 30 held-out tasks with graders, runnable with one command, 3 seeds each.
4. BASELINE.md: original agent completion, cost, latency with intervals.
5. The rebuilt agent: context/ (budgeter, tool-output compactors, retrieval), memory/ (schema, store, retrieve, update, forget), harness/ (loop, step cap, retries, verification, guardrails), tools exposed via MCP.
6. ABLATION.md: original, +context, +memory, +harness, all three - completion, cost, latency.
7. RETRIEVAL.md: Onyx vs a simple baseline retriever on retrieval questions from the eval.
8. CLAIMS.md: Meta-Harness 6x claim checked against the paper; Kimi K2.5 as backend tested only if time allows and only as a separate arm.

BUILT-IN GAP ANALYSIS: Maintain GAPS.md: failure classes not addressed, tasks still failing after rebuild and why, layers with no measurable effect, grader weaknesses, claims not verified. Fill the top items before finishing.

SELF-QUESTIONS: What share of failures were truly model-limited? Did I hold the model and temperature fixed? Are my graders checking the outcome, or the transcript's claims? Does memory retrieve the right item, and does it ever retrieve a stale one? Did compact tool outputs drop information the model needed? Did the harness's verification step catch real errors or only add cost? Would the gains survive a different task distribution?

STOP CONDITIONS: Stop when ABLATION.md has all five variants at 3 seeds and GAPS.md is written. Stop and ask if the baseline completion is already above 90% (the eval is too easy) or below 10% (likely a harness bug).

RETURN FORMAT: Failure taxonomy table, ablation table with intervals, retrieval comparison, the Meta-Harness claim verdict, and GAPS.md.

Pick one agent you own (a Claude Code workflow, an Agent SDK app, or a custom loop). Add tracing with OpenTelemetry or Langfuse so each run records: system prompt, tool schema size, history, retrieved content, tool outputs - each with its token count - plus tool calls, errors, retries, final output and wall time. Run it on 50 real tasks from your actual use (not synthetic). Save traces/ as JSONL. Also read the raw sources: the Meta-Harness paper and repo (record the benchmark, baseline and conditions behind the 'up to 6x' figure), Richmond Alake's memory material for the memory operations taxonomy, Anthropic's posts on building effective agents and context engineering, and the onyx-dot-app/onyx README and retrieval code. Write sources/NOTES.md with the claims each source makes and the evidence it gives.

Read every failed or degraded run in traces/. Classify each into one primary cause: context (needed info missing, buried, or crowded out; tool output too large or unreadable), memory (lost facts from an earlier session, stale or contradicting memory), harness (loop stuck, no retry on tool error, no verification, wrong stopping, unsafe action), or model (reasoning genuinely beyond capability with perfect inputs). Write FAILURES.md with counts and 2 trace ids per class. Then build eval/: 30 tasks, at least 5 per class, each with a deterministic grader (unit test, file diff, API state check, exact field match). Hold out 10 tasks you will not look at while building. Make it runnable with `python -m eval.run --variant <name> --seeds 3`. Store the held-out 10 in a separate file.

Pin model id, temperature and max tokens in eval/config. Run the original agent on all 30 tasks at 3 seeds. Record per task: pass/fail, tokens in and out, cost, wall time, number of tool calls, and the failure class if it failed. Compute completion rate with a 95% interval (Wilson), median cost and latency. Write BASELINE.md. If completion is above 90% the suite is too easy - add harder tasks from traces; if below 10% check for a harness bug before proceeding. Break failures down by class from FAILURES.md so later layers can be judged on the classes they target: a context fix should move context-class tasks, a memory fix should move the cross-session tasks. Also record how many failures were silent, meaning the agent declared success but the grader failed it; that number is the harness's honesty baseline and the verification step in step 6 must reduce it. Save raw per-run records to eval/results/baseline/.

Write context/budget.py that allocates a per-turn token budget by section (system, tools, memory, retrieval, history, tool output) and trims by priority. Replace raw tool outputs with compactors designed for the model: structured summaries with the fields the next step needs, pagination handles for detail, error messages that say what to try next - the angus.sewell AI-first data layer idea. Replace 'dump everything' with retrieval: implement a simple baseline (SQLite FTS5 or pgvector) and, separately, deploy Onyx (docker compose from onyx-dot-app/onyx) with your docs connected; compare both on the eval's retrieval-dependent tasks and write RETRIEVAL.md (recall at 5, answer accuracy, latency, ops cost). Reduce the tool schema to tools actually used in traces. Run the eval as variant +context and log tokens by section.

Write memory/ with three stores: episodic (what happened, with timestamps and task ids), semantic (facts about the user, project and environment, each with source and last-verified date), procedural (skills or playbooks that worked, with success counts). Implement operations: store (after task end, extract candidate memories, deduplicate), retrieve (by query with recency and relevance scoring, max N items into the context budget), update (on contradiction, keep newer with source, keep a history), forget (expire unverified facts after a set age; delete on user request). Persist to SQLite or Obsidian markdown files if you follow chase.h.ai's pattern. Run the eval as +memory alone and then +context+memory, including the memory-class tasks that span two sessions. Cap memory items injected per turn and log which were retrieved.

Write harness/: a loop with a step cap and token cap per task; retry with backoff on transient tool errors and a different approach on repeated errors; a planning step for tasks over N tool calls; a verification step that runs the task's own checks (tests, a read-back of the changed state) before declaring done; confirmation gates on destructive tools; structured final output. Expose tools through MCP servers with typed inputs and compact outputs. Read the Meta-Harness repo and, if feasible, run its approach for a few iterations on your harness using your traces and eval (full code plus logs as input, not summaries); otherwise record why not. Run the eval as +harness and as all three layers, then compare silent-failure counts to baseline.

Run the 10 held-out tasks for the first time on baseline and all-three variants, 3 seeds each. Build ABLATION.md: original, +context, +memory, +harness, all three - completion with Wilson intervals, cost, latency, per failure class. If time allows, run a 20-task sample of a public benchmark (tau-bench or SWE-bench Verified) on original vs rebuilt for calibration. Write CLAIMS.md: the Meta-Harness 'up to 6x' figure - what benchmark, what baseline, and whether your result is consistent; whether a model swap (e.g. Kimi K2.5 as a separate arm) would have closed the same gap. Write GAPS.md: failure classes still open, tasks still failing with reasons, grader weaknesses, layers with no effect. Rank and fix the top two, then re-run only the affected tasks.

You did not build this agent. Audit it. Read FAILURES.md, BASELINE.md, RETRIEVAL.md, ABLATION.md, CLAIMS.md, GAPS.md and the eval/ code. Enumerate every claim: failure class counts, completion rates, intervals, cost changes, retrieval metrics, the Meta-Harness verdict. Mark each VERIFIED (re-ran and matched within seed variance), UNVERIFIED (could not re-run; why), or FALSE (different result; show both). Re-run the held-out 10 tasks on the all-three variant at 3 seeds. Check the model id is identical across variants in result files. Test graders against a deliberately wrong output for 5 tasks. Plant a stale memory and see if it wins. Try a destructive tool without confirmation. Check traces and memory stores for secrets or personal data. List missing pieces (no eval in CI, no memory size cap, no cost alarm, retrieval index not refreshed, no human-in-loop path), rank by how likely each is to cause a production failure, fix the top three now, and report what you could not fix and why.