An AI Evaluation Set Needs the Cases Your Demonstration Avoids

A useful evaluation includes missing information, conflicting instructions and difficult exceptions, so the test measures the work a deployment will actually face.

A demonstration selects an experience

A demonstration usually has a purpose: to show what a tool can do. An evaluation has a different purpose: to discover how well it serves a defined task, including the conditions under which its output becomes unreliable or unusable. Confusing these activities can turn an attractive presentation into an unsupported operating decision.

NIST's 2024 Generative AI Profile provides a voluntary resource for considering generative AI risks across the lifecycle. The discussion here is an editorial application to evaluation design. It is not a new benchmark report, a claim about the performance of any named model or a certification method endorsed by NIST. The practical question is which cases deserve to influence a local deployment decision.

Define success before collecting examples

A test case needs a standard against which the result can be assessed. A response can be fluent and still omit information essential to the task. It can also differ from a preferred draft while being entirely acceptable. The evaluation should distinguish these situations rather than treating similarity to one sample answer as the whole measure of quality.

For example, a fictional support task may require recognising a missing order detail and asking for it before recommending a next step. The best answer is not necessarily the longest or most confident one. It is the response that respects the available evidence and advances the task appropriately. Writing that criterion beforehand makes the test less dependent on a reviewer's immediate impression.

Include incomplete inputs

Real requests often omit something important. A system should be tested on whether it identifies the gap, proceeds within a safe boundary or invents the missing information. An evaluation consisting only of complete, well-structured inputs cannot establish how the tool behaves when that assumption fails.

Create examples reflecting the actual kinds of omissions the team encounters. Remove a required date, leave a field ambiguous or supply a document whose relevant section is absent. The objective is not to trick a system for entertainment. It is to determine whether the deployment can handle a foreseeable operating condition without giving users an unwarranted impression that the task has been completed.

Test conflicting information separately

A missing fact and a contradictory fact are different problems. A system that can request an absent number may still choose arbitrarily between two incompatible values. An evaluation should examine whether the response preserves uncertainty and directs attention toward the conflict.

Imagine two versions of a fictional delivery record with different dates. The expected behaviour might be to flag the discrepancy and identify which source the business normally treats as authoritative. That rule must be supplied by the operation; it should not be invented by the evaluator after seeing the result. The test then examines both the system's response and whether the surrounding organisation has established a usable standard.

Use exceptions with consequences

A sample dominated by routine cases may describe routine performance well while offering little information about a small category of consequential failures. The evaluation should identify which exceptions matter because of their effect, not merely because of their frequency.

This does not mean allowing rare edge cases to obscure all useful performance. Report the groups separately. A tool may perform well enough for one part of the workflow and need additional review for another. That finding can support a narrower deployment instead of a binary argument about whether the technology works. The operating decision becomes more accurate when the test preserves these differences rather than reducing everything to one average score.

Keep a test from becoming its own training script

If the same examples are repeatedly used to improve an implementation, later results may describe familiarity with that set rather than broader reliability. Maintain a distinction between material used for development and material reserved for assessment. Protect sensitive information when constructing both.

A business also needs a process for adding newly discovered failure cases. The test set should evolve as the workflow changes, but changes should be documented so that comparisons remain interpretable. Otherwise an apparent improvement may partly reflect an easier collection of examples. A meaningful evaluation record explains what changed in the tool, what changed in the test and what remained stable.

Let disagreement improve the standard

Two qualified reviewers may disagree about an output. That disagreement can reveal an ambiguous task definition rather than a defective reviewer. Preserve the case and ask which criterion would resolve it. If the business cannot establish an acceptable response, the uncertainty should remain visible in the assessment. Do not manufacture agreement merely to produce a clean score. A useful evaluation can distinguish a clearly wrong answer from a disputed judgment, and that distinction helps management decide where automated assistance needs an accountable person with relevant expertise.

Judge the whole operating response

An evaluation should include what people do with the output. If a system flags uncertainty in a way that users routinely overlook, the deployed process may remain unreliable. Observe whether reviewers understand the signal and whether they have a practical route to resolve the issue.

My conclusion is that difficult cases belong at the centre of a serious assessment. They reveal the conditions under which assistance needs limits, escalation or a different design. A good evaluation does not merely support a purchasing argument. It gives the organisation an account of where a tool is useful, where evidence remains insufficient and what must happen when ordinary assumptions no longer hold.

Gabriel Nicolaev — CyGuru

Sources & further reading

NIST — Generative AI Profile, published 26 July 2024 ↗

Editorial photograph for illustration.

FOLLOW THE CONNECTION

Another angle on this subject

Continue through related ideas, decisions and consequences.

Technology & AI

An AI Supplier Update Is an Operating Change, Not Just a New Feature

Procurement needs a plan for changed model behaviour, revised integrations and rollback before an AI service becomes embedded in essential work.

Technology & AI

AI Productivity Is a Workflow Question

An impressive demonstration cannot tell a business whether its whole operation will become faster, better or cheaper.

Technology & AI

An AI Agent Should Not Inherit Every Permission Its User Holds

The next operating question for AI agents is how much authority each task actually requires, and how a mistaken action can be contained.