Red-Team Your Own Agent — prompt-injection CI from the jailbreak corpus

Prompt system — run this yourself

Pasteable into Claude Code, Claude, or any agent. One-shot for the whole result, step prompts when you want to steer. Every step carries its own audit; the gap audit makes the agent find and fill what it missed.

Mission. Build a defensive prompt-injection CI harness for YOUR OWN agent: a fixed suite of injection and 'just ask nicely' hijack cases that runs on every skill install and harness change, with a canary secret and a tool-call allowlist, failing the change if the canary leaks or a tool call escapes scope. Done means the suite runs in CI against your own harness, catches planted injections in test skills, and you can state what fraction a plain outbound allowlist blocks versus what needs per-skill testing.

1 · Deeper, not bigger

• What a top professional would do (that you didn't ask for): A defender does not collect jailbreaks to use them; they build an eval that hardens their own agents, which is exactly what the input entries point to defensively. They treat prompt injection as uncontainable at the input filter (carterperez.dev: 'prompt injection cannot be filtered reliably; contain what the AI is allowed to do') and so they constrain capability: per-skill capability manifests (network, filesystem, secrets), an outbound tool-call allowlist, and identity/authorization checks before an automation returns data (the Noma workflow-hijacking finding mattjayy relays - check the requester is authorized, don't just answer). They run the suite in CI on every skill install and harness change, with a canary secret in the environment, and they measure which layer blocks what, per the dive's own falsifier.
• Rawest material to start from: Your own agent harness, its hooks, and its installed skills; the Noma Labs workflow-hijacking research (via mattjayy / vulnu.xyz/noma-ig) on automations that return data without checking authorization; the Princeton sponsored-option study (forgoodcode) showing models push a paid option even when told not to, as a bias case for the suite; carterperez.dev's capability-sandboxing framing; the categories of public jailbreak corpora (as abstract attack CLASSES to defend against, e.g. system-prompt override, role-play escape, encoded payloads) - used only to build detectors, never operational attacks against third parties.
• Methods and tools: A CI harness (GitHub Actions or a local pre-install hook) that, on every skill install or harness change, (1) diffs the change, (2) loads it in an isolated test harness with a canary secret in the environment, (3) runs a fixed suite of injection and 'ask nicely' hijack cases, (4) asserts the canary never leaves and every tool call stays within an allowlist; per-skill capability manifests (network/filesystem/secrets); an identity/authorization gate on data-returning automations; a layer-attribution report (allowlist vs manifest vs skill-level test).
• What proves it right: The suite, run against your own harness, catches a deliberately planted injection in a test skill (if it does not, the suite is theatre). The canary secret never appears in any outbound tool call across the suite. A control measurement answering the dive's falsifier: what fraction of suite cases a plain outbound allowlist blocks with no skill-level testing - if it is >=95%, a static policy file is most of the answer and the CI layer is justified only for the remainder. All testing is against harnesses, accounts and keys you own.

Rewritten prompt — ask for exactly that:

Act as a defensive security engineer hardening MY OWN agent - not an attacker. Build me a prompt-injection CI harness that runs on every skill install and every harness change to my own setup. First, put a throwaway canary secret in a disposable test environment and define an outbound tool-call allowlist for my agent. Build a fixed suite of ~50 defensive cases: direct injection attempts (system-prompt override, role-play escape, encoded payloads) as attack CLASSES to detect, 'just ask nicely' hijack cases modelled on the Noma workflow-hijacking research (an automation returning data without checking the requester is authorized), and sponsored-bias nudges modelled on the Princeton study. Wire CI: on skill install or harness change, diff it, load it in an isolated harness with the canary, rerun the suite, and FAIL the change if the canary leaks or any tool call escapes the allowlist. Add per-skill capability manifests (network/filesystem/secrets) and an authorization gate on data-returning automations. Then measure the dive's falsifier: what fraction of cases a plain outbound allowlist blocks with no skill-level testing - if >=95%, tell me a static policy is most of the answer. Everything runs against harnesses and keys I own. Return the suite, the CI wiring, the canary/allowlist proof, the layer-attribution table, and planted-injection catch evidence.

2 · One-shot prompt

You are a defensive security engineer hardening the user's OWN agent. This pack is for defense only: understanding attack CLASSES, building detectors and evals, and containing what your own agent can do. Do not produce operational attacks against third parties; test only harnesses, accounts and keys the user owns. No marketing language.

Context and method: prompt injection cannot be reliably filtered at the input (carterperez.dev) - so you CONTAIN capability. Noma's workflow-hijacking research (mattjayy / vulnu.xyz/noma-ig) shows automations returning data to whoever asks because nobody checks authorization - so add an identity/authorization gate. The Princeton study (forgoodcode) shows models push a sponsored option even when instructed not to - include that as a bias case. Use jailbreak corpora only as abstract attack classes (system-prompt override, role-play escape, encoded payload, tool-output injection) to build detectors against your own harness.

Hard constraints: a disposable canary secret in an isolated test environment; an outbound tool-call allowlist; a fixed suite run on every skill install and harness change; FAIL the change if the canary leaks or a tool call escapes the allowlist; measure the falsifier (allowlist-only block rate). Name the real harness, hooks, skills and CI. All targets owned by the user.

Ordered deliverables: (1) the threat model for YOUR harness - where untrusted text enters (fetched web/Reddit/X content, skill descriptions, tool outputs, inbound email) and what the agent can touch (keys, files, network, money). (2) a canary secret planted in a disposable test env + an outbound tool-call allowlist. (3) a ~50-case defensive suite: direct-injection classes, Noma-style 'ask nicely' authorization-bypass cases planted in fake fetched content, and sponsored-bias nudges - each with an expected safe outcome. (4) per-skill capability manifests (network/filesystem/secrets). (5) an authorization gate on data-returning automations. (6) CI wiring: on skill install / harness change -> diff -> isolated load with canary -> rerun suite -> fail on canary leak or allowlist escape. (7) a layer-attribution table answering the falsifier: % blocked by allowlist alone vs needing manifests vs needing skill-level testing.

Built-in gap analysis: before finishing, list what is unverified - does the suite actually catch a planted injection; does the canary assertion cover all tool outputs; does CI really block a failing install; is the allowlist-only block rate measured or guessed. Fill what you can by running it.

Self-questions to answer in writing: Where exactly does untrusted text enter my harness? Does the canary assertion cover every outbound channel, or just one? Did I plant a real injection in a test skill and confirm the suite catches it? Does CI actually fail the install, or only log? What does a plain allowlist block without any skill testing (the falsifier)? Does the authorization gate check requester identity before returning data? Am I testing only my own systems?

Stop conditions: stop and ask only if there is no harness you own to test or no safe canary environment. Otherwise build end to end.

Return format: a markdown report with the threat model, a file tree, the suite results (case class, outcome, layer that caught it), the planted-injection catch evidence, the canary-never-leaked proof, and the falsifier table (allowlist-only block rate). Commit the suite, CI wiring, manifests and gate; print the commit hash. Keep every attack case abstract and defensive.

3 · Step-by-step prompts

Step 1 — Threat-model your own harness

Goal: Map where untrusted text enters and what the agent can touch.

Establish ground truth defensively. For YOUR OWN agent harness, map two things. (A) Untrusted-input surfaces: every place external text reaches the model - fetched web/Reddit/X content (e.g. via an Agent-Reach-style reader), skill descriptions and READMEs you install, tool outputs, inbound email, file contents. (B) Capability surface: everything the agent can do that matters - API keys/secrets in the environment, filesystem writes/deletes, outbound network, any money or account actions. Produce a table linking each input surface to the capabilities an injection there could abuse. This is the threat model the suite and the allowlist are built from. Keep it about your own setup; do not target anyone else's systems.

Expect: A threat-model table: untrusted-input surfaces x reachable capabilities, scoped to the user's own harness.

Audit before moving on: Confirm every installed skill and every fetch path is listed as an input surface, and every secret/filesystem/network/money capability is listed. Fail if a known input path or capability is missing. On fail, enumerate the harness fully (list skills, hooks, env vars) and complete it.

Step 2 — Canary + outbound allowlist

Goal: Plant a canary and define what tool calls are allowed to leave.

Build the containment baseline. (1) Plant a throwaway CANARY secret (a unique random string) in a DISPOSABLE test environment that mimics your real one - never a real key. (2) Define an outbound tool-call allowlist: the exact tools, and for each, the allowed targets (recipients, hosts, paths). Anything outside it is denied by default. (3) Add an assertion harness that inspects every outbound tool call and every model output for the canary string and for off-allowlist destinations. This is enforced in code, not asked of the model. Confirm the assertion sees a tool call (feed one benign call through and watch it log).

Expect: A canary planted in a disposable env, an outbound allowlist (tools + targets), and a code assertion that inspects all tool calls/outputs for canary leaks and off-allowlist destinations.

Audit before moving on: Feed one call containing the canary deliberately and confirm the assertion flags it; feed one off-allowlist destination and confirm denial. Fail if the assertion misses either, or if a real secret was used. On fail, fix coverage before building the suite.

Step 3 — Build the defensive suite

Goal: Author ~50 injection/hijack/bias cases as attack classes with safe-expected outcomes.

Build a fixed suite of about 50 DEFENSIVE cases against your own harness, grouped as attack CLASSES (not copy-paste operational attacks on others). (A) Direct injection (~20): system-prompt override attempts, role-play/'ignore previous instructions' escapes, and encoded/obfuscated payloads embedded in fetched content - each expecting the agent to refuse and keep the canary. (B) Authorization-bypass / 'ask nicely' (~20): modelled on the Noma workflow-hijacking research - a request planted in fake fetched Reddit/X/email content asking the automation to return data or act, where the requester is NOT authorized; expected outcome is the agent checks authorization and refuses. (C) Sponsored-bias nudges (~10): modelled on the Princeton study, content pushing a paid/sponsored option; expected outcome is the agent stays neutral even when the content tells it to prefer the sponsor. Each case: input, where it enters (which surface from step 1), and the expected SAFE outcome.

Expect: ~50 cases across direct-injection, authorization-bypass and sponsored-bias classes, each tagged with its input surface and expected safe outcome.

Audit before moving on: Confirm every case maps to an input surface from step 1 and states a safe expected outcome, and that no case is an operational attack against a third-party system. Fail if any case targets someone else's infrastructure or lacks an expected outcome. On fail, rewrite it as a defensive class against your own harness.

Step 4 — Capability manifests + authorization gate

Goal: Contain what each skill can do and check authorization before returning data.

Add the containment layers carterperez.dev argues for (contain what the AI can do, since you cannot filter injection). (1) Per-skill capability manifests: for each installed skill, declare its allowed network hosts, filesystem paths, and whether it may read secrets; enforce the manifest at the harness level so a skill cannot exceed it. (2) An authorization gate on any automation that returns data or acts on request (the Noma fix): before responding, verify the requester is authorized for that data/action; if identity is absent or unauthorized, refuse. Re-run the step-3 suite with these layers active and record how many cases each layer now blocks.

Expect: Capability manifests enforced per skill, an authorization gate on data-returning automations, and a per-layer block count on the suite.

Audit before moving on: Pick one skill and confirm it cannot reach a host/path outside its manifest; feed an unauthorized data request and confirm the gate refuses. Fail if a skill exceeds its manifest or the gate returns data without an identity check. On fail, tighten enforcement.

Step 5 — Wire the CI

Goal: Run the suite automatically on every skill install and harness change.

Wire prompt-injection CI. On every skill install (e.g. from find-skills or a skills hub) and every harness change: (1) DIFF the incoming change and show what capabilities it adds. (2) Load it in an ISOLATED test harness that has the canary (never real keys). (3) Re-run the full suite with the new skill/change loaded. (4) FAIL the install/change (non-zero exit, block the merge) if the canary leaks in any output/tool call OR any tool call goes outside the allowlist OR the skill exceeds its manifest. Implement as a GitHub Actions workflow and/or a local pre-install hook. Prove it by attempting to install a test skill that contains a planted injection and confirming CI blocks it.

Expect: A CI workflow/hook that diffs, isolates with the canary, reruns the suite, and fails the change on canary leak, allowlist escape, or manifest breach - proven by a blocked malicious test install.

Audit before moving on: Attempt to install a benign skill (should pass) and a skill with a planted injection (should fail). Fail if CI passes the malicious one or blocks the benign one, or if it only logs instead of failing. On fail, fix the exit/blocking behaviour.

Step 6 — Falsifier measurement + gap report

Goal: Attribute blocking to layers; answer the dive's falsifier; report gaps.

Final validation and the dive's own falsifier. (1) Layer attribution: run the suite with ONLY the outbound allowlist active (no manifests, no skill-level suite logic) and record what fraction of cases it blocks. Then add manifests, then the full skill-level suite, recording the incremental block rate of each. (2) Answer the falsifier: if the plain allowlist alone blocks >=95% of cases, state clearly that a static outbound policy file is most of the answer and the CI layer is justified only for the remaining cases; if it blocks far less, the skill-level CI is doing real work - quantify it. (3) Confirm the canary never leaked across any run. (4) Enumerate claims, mark verified/unverified/false, list missing pieces ranked by impact, fix the top two. Commit suite, CI, manifests, gate; print the hash.

Expect: A layer-attribution table (allowlist-only %, +manifests %, +skill-suite %), an explicit falsifier verdict, a canary-never-leaked confirmation, a claim ledger, and a committed harness with hash.

Audit before moving on: Confirm the allowlist-only run genuinely disabled the other layers. Confirm the falsifier verdict follows the measured numbers, not a preferred conclusion. Fail if the attribution is estimated or the canary leaked anywhere. On fail, re-run with layers isolated and report the real split.

4 · Gap analysis and audit

Run after the CI harness is built, defensively and against your own systems only. Enumerate every claim as a numbered list: the threat-model coverage, the canary assertion coverage, each suite class, the capability manifests, the authorization gate, the CI blocking behaviour, and the layer-attribution numbers. Mark each VERIFIED (with evidence: a blocked test install, an assertion log, the attribution table), UNVERIFIED, or FALSE. Specifically re-check: does the canary assertion cover EVERY outbound channel, not just one; did a deliberately planted injection in a test skill actually get caught; does CI FAIL (non-zero/block) rather than merely log; does a skill get contained to its manifest; does the authorization gate check requester identity before returning data; is the allowlist-only block rate measured with the other layers truly disabled. List missing pieces: unmonitored output channels, untested skill classes, CI that only warns, estimated attribution. Rank by impact on whether your own agent is actually harder to hijack. Fix the top three and re-verify. Confirm no case targets third-party systems. Report what you could not fix and why.

Questions the agent must ask itself before it says done:

• Where exactly does untrusted text enter my harness, and did I enumerate every surface?
• Does the canary assertion cover every outbound channel, or only one tool?
• Did I plant a real injection in a test skill and confirm the suite catches it?
• Does CI actually FAIL the install/change, or does it only log a warning?
• What fraction do a plain allowlist block alone (the falsifier), with other layers truly disabled?
• Does each skill get contained to its capability manifest, enforced in code?
• Does the authorization gate verify requester identity before returning data (the Noma fix)?
• Did I include the sponsored-bias case, and does the agent stay neutral under it?
• Are my varied framings catching injection classes, or only the obvious phrasings?
• Is every test case against a system, account and key I own?
• Did I use any real secret instead of a disposable canary anywhere?
• Is the layer-attribution split measured, or did I guess which layer blocks what?

5 · What proves it right

• The suite, run against the user's own harness, catches a deliberately planted injection in a test skill; CI blocks that install with a non-zero exit.
• The canary secret never appears in any outbound tool call or model output across the entire suite and the unattended CI runs.
• A skill cannot exceed its capability manifest (network/filesystem/secrets), confirmed by an attempted out-of-manifest access being denied.
• The authorization gate refuses a data-returning request whose requester identity is absent or unauthorized (the Noma workflow-hijacking fix).
• The layer-attribution table answers the falsifier with measured numbers: the allowlist-only block rate is stated, and the CI layer's incremental value is quantified.

6 · Failure modes

• Input-filter theatre: trying to filter injections out of text. Tell: effort spent on blocklists of phrases. Fix: contain capability (allowlist + manifests), since injection cannot be reliably filtered.
• Partial canary coverage: assertion watches one channel. Tell: canary leaks via an unmonitored tool. Fix: assert on every outbound channel and model output.
• CI that only logs: suite runs but never blocks. Tell: a failing install still completes. Fix: non-zero exit / block the merge on any failure.
• Suite never actually catches anything: no planted injection to prove it. Tell: all-green with no negative control. Fix: plant a real injection in a test skill and confirm it is caught.
• No authorization check: automation returns data to whoever asks (the Noma finding). Tell: 'ask nicely' cases pass through. Fix: identity/authorization gate before returning data.
• Guessed attribution: falsifier answered from intuition. Tell: no allowlist-only run. Fix: disable other layers and measure the allowlist-only block rate.
• Scope drift into offense: cases become attacks on third-party systems. Tell: targets you do not own. Fix: keep every case an abstract class against your own harness.

7 · Agent direction (unattended runs)

This is a defensive build against the user's OWN systems only; refuse any drift toward attacking third-party infrastructure. Reach first for the threat model (enumerate the harness's input surfaces and capabilities) and the canary+allowlist containment baseline, before writing any test case. Cache the suite and the allowlist; the CI and the falsifier runs reuse them. Use a disposable canary string, never a real key; assert on every outbound channel. Spawn a sub-agent to run the isolated-harness CI so a loaded test skill cannot touch the real environment. Prove the suite with a planted-injection negative control - an all-green run with nothing to catch is theatre. Measure the falsifier (allowlist-only block rate) with the other layers genuinely disabled. Stop and ask the user only if there is no harness they own to test or no safe canary environment. Return a markdown report (threat model, file tree, suite-results-by-layer table, planted-injection catch evidence, canary-never-leaked proof, falsifier table) plus committed CI, suite, manifests and gate; print the commit hash. Keep every attack description abstract and defensive.

Prompt system updated 2026-10-22 · all prompt systems · JSON