Guardrails and evals for AI agents: what they are and how to know the agent got it right before production

Guardrails and evals answer different questions. A guardrail asks, at every action: "may this happen now?". An eval asks, across hundreds of runs: "does this agent get it right, and does it keep getting it right after the last change?". An agent ready for production needs both. If you treat one as the other, you ship without knowing whether the agent works.

Note: "harness" here is the software concept, the infrastructure around the model. If the term is new to you, start with what is an AI harness.

Where this post fits

On this blog we have already explained harness engineering, how to build a harness from failures and why Claude Code is a harness. What was missing is how you know the harness works. Mitchell Hashimoto's loop of turning errors into rules assumes you can see the error. Guardrails and evals show you the error before the deploy.

Guides and sensors: Böckeler's taxonomy

Birgitta Böckeler (Thoughtworks) gives the vocabulary in "Harness engineering for coding agent users" (martinfowler.com, April 2, 2026):

• Guides (feedforward) anticipate the agent's behavior and try to steer it before it acts.
• Sensors (feedback) observe after the agent acts and help it correct itself.

She adds a second axis, which matters most for evaluation:

• Computational. Deterministic and fast: tests, linters, type checkers, structural analysis.
• Inferential. Semantic analysis, AI review, "LLM as judge". Slower, more expensive and non-deterministic.

These two axes let you classify guardrails and evals. A guardrail is a sensor (sometimes a gate) that runs inside the agent loop, on every action. An eval is a sensor that runs outside the loop, over a set of tasks, to measure aggregate behavior.

Guardrails: the sensor that runs before the action

A guardrail does not measure quality. It enforces a limit: "never run git push --force", "stop everything when I say so".

A good guardrail is computational whenever possible. A rule in the prompt ("do not delete files") is an inferential guide. It helps, but it degrades as the context fills up. A hook that blocks the call works the same in the first minute and in the tenth session. That is the principle behind capivaOS: a rule that must always hold becomes a hook, not a sentence in the prompt.

Claude Code Mods: guardrails as TypeScript code

On October 1, 2026, Anthropic launched Claude Code mods: small TypeScript (or JavaScript) functions that run inside Claude Code and attach to the events it emits. For anyone putting an agent into production, the event that matters is tool.call.

According to the official docs, the tool.call event fires every time Claude is about to run a tool, including in subagents and MCP tools. The hook can:

• Observe. Log and return next(e), and the tool runs.
• Rewrite. Call next with changed arguments.
• Answer instead. Return { deny: reason } without calling next. The tool does not run, and Claude reads the reason as the tool result.

In Böckeler's taxonomy, a blocking hook on tool.call is a computational sensor placed before execution. It observes the action the agent has already decided to take and returns a signal the model can read. Böckeler herself describes sensor messages written for the LLM to consume as "a positive kind of prompt injection". The deny text is exactly that, and the docs recommend writing it as an instruction Claude can follow.

A hook on prompt.submit or prompt.section, which rewrites what the model reads, is a guide instead. It acts before the agent decides anything.

Example: a kill switch in a file

The simplest example is a switch that stops every tool when .claude/KILL exists in the project. A human (or an alert) creates the file, and the agent stops on its next call.

The code below follows the documented API: on('tool.call', ...), $.fs.exists (which resolves to boolean in the official type declarations) and .catch to fail closed. Anthropic warns that events can change between versions and that the types generated for your build take precedence over any page, so check against your version (mods require Claude Code v2.1.287 or later in the terminal).

kill-switch/
├── .claude-plugin/plugin.json
└── hooks/
    ├── hooks.json        → { "modules": ["./register.ts"] }
    └── register.ts

// kill-switch/hooks/register.ts
import type { On } from 'claude-code'

// Relative path: resolves against the session's working directory
const KILL_FILE = '.claude/KILL'

export function register(on: On) {
  // No matcher: runs before EVERY tool call,
  // including subagents and MCP tools
  on('tool.call', async ($, e, next) => {
    if (await $.fs.exists(KILL_FILE)) {
      // Does not call next(): the tool does not run and Claude reads the reason
      return {
        deny:
          'Kill switch active (.claude/KILL exists). Stop the task, ' +
          'do not try another tool and wait for instructions from a human.',
      }
    }
    return next(e)
  }).catch(async ($, e, next) => {
    // Without this handler, a hook that fails before next() is skipped
    // and the call goes ahead (fail open). Here, we fail closed.
    if (next.called) return next(e)
    return { deny: 'Kill switch unavailable (' + next.error.kind + '). Call blocked for safety.' }
  })
}

The guardrail gets tests too. The claude-code/testing kit fires events with no session, no login and no network, with stubs in place of the mods API calls (Test a mod):

// kill-switch/tests/kill-switch.test.ts
import { expect, test } from 'claude-code/testing'

test('blocks every tool when the KILL file exists', async ($, on) => {
  on('fs.exists', () => ({ value: true }))
  const r = await $.tool.call({ tool: 'Bash', command: 'ls' })
  expect(r.deny).toContain('Kill switch active')
})

test('lets the call through when the file does not exist', async ($, on) => {
  on('fs.exists', () => ({ value: false }))
  on('tool.call', () => ({ result: 'ok' }))
  const r = await $.tool.call({ tool: 'Bash', command: 'ls' })
  expect(r.result).toBe('ok')
})

test('fails closed when the file cannot be checked', async ($, on) => {
  on('fs.exists', () => ({ deny: 'EACCES' }))
  const r = await $.tool.call({ tool: 'Bash', command: 'ls' })
  expect(r.deny).toContain('Kill switch unavailable')
})

The third test simulates $.fs.exists failing. A stub that returns { deny } makes the call reject inside the mod, as the docs describe. I ran the two files above with Claude Code 2.1.296 on Linux, with no login and no API key, from the kill-switch/ folder. The run used Portuguese test names; they are translated below:

$ claude plugin validate .
...
Validating hooks: .../kill-switch/hooks/hooks.json

  ❯ ./register.ts hooks: tool.call
  ❯ ./register.ts gating hook with .catch: tool.call
  ❯ ./register.ts calls: $.fs.exists

✔ Validation passed with warnings

$ claude plugin test

tests/kill-switch.test.ts:
(pass) blocks every tool when the KILL file exists [47.36ms]
(pass) lets the call through when the file does not exist [14.61ms]
(pass) fails closed when the file cannot be checked [12.10ms]

 3 pass
 0 fail
Ran 3 tests across 1 file. [0.20s]

The only validate warning was the missing author field in plugin.json. To watch the fail-open happen, I deleted the .catch from register.ts and ran it again. The third test failed, and the engine shows why:

(fail) fails closed when the file cannot be checked [15.84ms]
  HooksError: no implementation for tool.call
  ...
  the engine reported:
    kill-switch's tool.call hook was skipped: kill-switch: $.fs.exists: EACCES

Without the .catch, the hook was skipped and the call went ahead. In the test there was nothing underneath to answer it, hence the no implementation. In a real session, the tool would have run. claude plugin test exited with status 1 in this case, so it runs in CI like any test suite.

The limits the docs make clear

An honest guardrail comes with a list of what it does not cover. According to the docs:

• Mods do not run in a sandbox. They have your user's permissions. Install them only from trusted sources and review them with claude plugin validate.
• Without .catch, the failure is open. A hook that throws or times out before next is skipped. The .catch itself has a 1 second limit; if it fails, the hook is skipped too.
• Tools executed by the API server (like the advisor) do not fire tool.call.
• Corporate policy comes first. PreToolUse hooks from managed settings run before any mod, and their block is final. Mandatory rules live there.
• Matching command text is a reminder, not security. To really protect main, protect it on the Git host.

A guardrail limits the possible damage. It does not tell you whether the agent does good work. That is what an eval is for.

Evals: the sensor that measures the whole agent

Anthropic defines it in "Demystifying evals for AI agents" (January 2026): an eval is a test for an AI system. You give it an input, apply grading logic to the output and measure success.

For agents, the system under evaluation is the model, the harness, the tools and the environment. Grade the outcome in the environment when you can (did the test pass? does the record exist?) rather than the final text, because the agent can announce it finished without finishing.

(The "eval harness", the runner that executes and scores, is not the "agent harness". The distinction is in AI harness and benchmarks.)

Offline and online

• Offline evals. They run before the deploy, over a fixed set of tasks, with no real user. They are what runs in CI on every change to a prompt, a tool or the model.
• Online evals. They run on production traffic: monitoring, samples of transcripts graded automatically, user feedback, A/B tests.

Anthropic uses the Swiss cheese image: no single layer catches everything. Automated evals are the first line in CI/CD. Production monitoring catches drift and unexpected failures. Periodic human review calibrates the rest.

Golden set: start with a few tasks, but real ones

A golden set is the curated set of tasks, each with an unambiguous success criterion, that represents what the agent has to get right. The common mistake is putting it off because "we need hundreds of cases". Anthropic recommends starting with 20 to 50 simple tasks taken from real failures.

A quality test for each task: would two experts reach the same verdict on their own? If not, the problem is the task, not the agent.

Three kinds of grader

The same article separates three kinds of grader, which map directly onto Böckeler's axes:

LLM-as-judge: useful, as long as it is calibrated

The reference paper is "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (Zheng et al., 2023). The authors show that strong judges (GPT-4, in the study) reached over 80% agreement with human preferences, the same level of agreement as between humans. They also document three biases: position (preferring the first answer), verbosity (preferring the longest one) and self-enhancement (preferring their own answers).

In practice, that becomes engineering rules:

• Swap the order of the answers and only declare a winner when the verdict holds in both orders.
• Use structured rubrics and grade each dimension with an isolated judge, as Anthropic recommends.
• Give the judge a way out. Allowing "Unknown" when information is missing reduces hallucination.
• Calibrate against humans before trusting it, and again when you change the judge model.

Regression: the eval that protects what already works

Anthropic separates two kinds of suite. Capability evals ask "what can this agent do?" and should start with a low pass rate, because that is where you improve. Regression evals ask "does it still do everything it used to?" and should pass close to 100%. A drop there means something broke.

The cycle closes with Hashimoto's loop. Every production failure becomes a new task in the golden set, and capability tasks the agent has mastered "graduate" to the regression suite.

A metric managers need to understand: pass@k and pass^k

Agents are non-deterministic, so one run per task is not enough. Anthropic uses two metrics. pass@k is the chance of getting it right at least once in k attempts. pass^k is the chance of getting all k right. With a 75% success rate per attempt and 3 attempts, pass^3 is 0.75³, about 42%.

For an internal tool where one good answer is enough, use pass@k. For an agent that serves customers, use pass^k.

Guardrails vs. evals, side by side

Guardrails without evals give you an agent that causes no disaster but may deliver nothing. Evals without guardrails give you an agent that is right almost every time, with no control over the rest.

Checklist before going to production

1. A written success criterion that can be checked in the environment.
2. A golden set of 20 to 50 real tasks, each with an unambiguous verdict.
3. Code-based graders first. An LLM judge only where there is no verifiable outcome.
4. An LLM judge calibrated against a human sample, with order swapping and a rubric per dimension.
5. Several attempts per task, reporting pass^k when consistency matters.
6. A regression suite in CI, running on every change to a prompt, a tool or the model.
7. Deterministic guardrails for irreversible actions: deletion, push, payment, external sends.
8. A tested kill switch that fails closed, and someone who knows how to trigger it.
9. Corporate policy in managed settings, not in an extension a user installed.
10. Online monitoring defined: what to sample, who reads the transcripts and how often.
11. Every production failure becomes a new task in the golden set.

Frequently asked questions

Where Capiva comes in

That is what Capiva's Technical Validation phase does before the larger investment. It defines a measurable success criterion, builds the golden set with your real data, measures the agent and designs the guardrails the operation needs. You leave with a go or no-go decision based on measured numbers.

If you have an agent working in a pilot and cannot say whether it is ready for production, let's talk.