# Evaluating AI agents This guide walks through using Candor to evaluate the output of an AI agent against a rubric. It covers how to decompose a multi-criteria eval into a single study, how to pick the right task type, how to write a rubric that produces reliable labels, how content renders to your labelers, and how to preview cost before anything charges. The worked example is a research assistant agent that reads source documents and writes summaries — but the same pattern applies to any multi-source AI agent: research assistants, customer support triage bots, GTM tools, code review agents, or anything else where you want human judgement on whether the output is good. **Scope.** This guide focuses on *item-based* studies (your agent produced some outputs, and you want humans to label them). For moderated interviews where humans talk to a Candor AI moderator about a product, see the [Studies](#studies) concept section above. ## Breaking down a complex eval Most real agent evals are multi-dimensional. You don't just want to know *"is this output good?"* — you want to know *why* it's good or bad across several orthogonal axes. Take a research assistant that reads a set of source documents and produces a summary. Four reasonable things to measure: - Did the agent retrieve the right source documents for the question? - For every claim in the output, is the source correctly attributed? - Is the summary faithful to what the sources actually say? - Does the output surface the most important findings — or bury them? The wrong instinct here is to run four separate studies, one per question. That costs 4× the labeler time (each labeler has to re-read the same output four times) and gives you four disjoint score streams that are hard to join. The right pattern is **one item per agent output, scored along multiple dimensions in a single pass**. Each labeler sees one output, answers all four questions on it at once, and moves on. In Candor this is a `scorecard` task with four `criteria`. > [!INFO] > If your dimensions are all orthogonal (measuring different things), fold them into one scorecard. If they aren't — if two of them are really asking the same thing — drop one. ## Picking a task type Candor supports five item-based task types. The choice is not about the mechanic — it's about the *question* you want answered. - *"Is this output good across several dimensions?"* — `scorecard` — Multiple weighted criteria, behavior-anchored levels, a weighted overall score. The right tool for almost every agent-eval scenario. - *"Is version A of my agent better than version B?"* — `compare (pairwise)` — Participant sees two outputs side-by-side, picks the winner. Use when absolute judgement is hard but relative is easy — e.g. comparing two model versions, two prompts, or two generation strategies. - *"Which category does this output fall into?"* — `label` — Participant picks one label from a fixed set. Good for taxonomies (e.g. intent classification, sentiment, failure-mode tagging). - *"On a 1–5 scale, how good is this?"* — `rate` — Single-dimension rating. Simpler than a scorecard, but you lose the per-dimension breakdown. Use only when you genuinely only care about one axis. - *"What did you notice? Freeform."* — `describe (free text)` — Participant writes an open-ended response. Great as a companion task to catch failure modes you didn't anticipate in your rubric — but hard to aggregate, so don't use it as your primary signal. For the research-assistant example — and for essentially any multi-criteria eval — **scorecard** is the right choice. The rest of this guide assumes scorecard. A quick word on pairwise, since it's the least obvious one: pairwise is useful when you're comparing two versions of something (two model variants, two prompt templates, two retrieval strategies) and absolute quality is subjective. People are much better at saying *"A is better than B"* than at saying *"A is a 4 out of 5"*. But pairwise only gives you relative rankings — it can't tell you whether any of the options are actually good. ## Designing a rubric A scorecard is a set of `criteria`. Each criterion has a `name`, a `weight`, and an ordered list of `levels` from worst to best. Good rubrics produce reliable labels; sloppy rubrics produce inter-rater disagreement and findings you can't act on. ### Keep dimensions to 3–5 More than five and labelers get fatigued; their later answers get sloppier. Fewer than three and you lose the whole point of a multi-dim eval. If you catch yourself wanting six dimensions, usually two of them are measuring the same thing. ### Use behavior-anchored levels A level description should tell a labeler *exactly* what they're looking for — ideally something they can observe or verify without having to make a judgment call. Compare: ```text BAD: "Summary is accurate" GOOD: "Summary correctly names every key point in the source documents" BAD: "Good attribution" GOOD: "Every claim in the output is traceable to a specific source document" BAD: "Useful" GOOD: "Surfaces at least 3 actionable findings with concrete next steps" ``` When two labelers look at the same output and pick different levels, it's almost always because the level wording was ambiguous — not because the output was genuinely borderline. Invest in the wording. ### Use 4 levels when you can An even number of levels forces labelers off the fence. Three or five levels give them a comfortable middle option ("Satisfactory", "Neutral") that they'll reach for whenever they're unsure — which means your middle bucket absorbs noise and tells you nothing. Four levels makes them pick a side. ### Weights are relative Weights only matter in ratio to each other. `[5, 3, 3, 1]` and `[10, 6, 6, 2]` produce identical overall scores. Use small integers and pick weights that reflect how you'd actually explain the decision to a stakeholder: "accuracy is way more important than formatting" becomes `weight: 5` for accuracy and `weight: 1` for formatting. Here's the criteria JSON for the research-assistant example: ```json [ { "name": "Source retrieval", "weight": 4, "levels": [ "Wrong sources entirely", "Missing critical sources", "Most sources retrieved with minor gaps", "All expected sources retrieved" ] }, { "name": "Source attribution", "weight": 3, "levels": [ "No attribution at all", "Some claims cited, others unsupported", "Most claims cited, one or two wrong", "Every claim correctly attributed to its source" ] }, { "name": "Summary accuracy", "weight": 5, "levels": [ "Inaccurate or fabricated details", "Partial or selective — omits key points", "Mostly accurate with minor detail errors", "Summary correctly names every key point in the sources" ] }, { "name": "Findings utility", "weight": 3, "levels": [ "Misleading findings that would confuse the reader", "Generic summary with no surfaced findings", "Some findings surfaced, no prioritization", "Clearly prioritized findings with next-step guidance" ] } ] ``` Note that levels are ordered *worst to best*, and accuracy has the highest weight because this team cares most about whether they can trust the summary. Your weights should reflect your own priorities. ## Content formats — what labelers actually see Before you prepare items, you need to know what your labelers will see on screen. Each item has three display slots: - `label` — short text. **Plain text only — no Markdown or HTML.** - `assetUrl` + `mimeType` — media. Supports `image/*`, `audio/*`, and `video/*` only. **Not HTML, not text files.** - `metadata.description` — optional small-gray caption rendered below the main content. Good for subtitles like "Q3 2026 run — v2.4 of the agent". When you set both `label` and `assetUrl`, both render — the media shows prominently and the label renders as a caption right next to it. It's not one or the other. Here's exactly what labelers see for each combination: **Image + label + description** ```diagram ┌──────────────────────────────┐ │ ┌──────────┐ │ │ │ │ │ │ │ IMAGE │ ← assetUrl (image/*) │ │ │ │ │ └──────────┘ │ │ item.label │ ← small caption below │ metadata.description │ ← smaller gray caption │ View product → │ ← if metadata.url └──────────────────────────────┘ ``` **Text only (no assetUrl)** ```diagram ┌──────────────────────────────┐ │ ┌────────────────────────┐ │ │ │ item.label │ │ ← in a gray bordered box │ └────────────────────────┘ │ │ metadata.description │ └──────────────────────────────┘ ``` Text-only items are fine for short strings (a headline, a single sentence, a filename) but not for anything longer. There's no wrapping control, no formatting, no line breaks — just a single styled span inside a gray box. A long paragraph will render as one ugly run-on. **Audio** ```diagram ┌──────────────────────────────┐ │ item.label │ ← label above the player │ ┌────────────────────────┐ │ │ │ ▶ ═══════════ 0:32 │ │ ← native