AI evaluation: build a useful test dataset
Build an AI evaluation dataset with realistic tasks, review criteria, source context and regression cases. Includes a practical worksheet and worked example.
A demo answers one question well. A product has to handle the questions people actually bring to it, including the incomplete ones, the awkward documents and the requests it should decline. An AI evaluation dataset gives the team a repeatable way to examine that work before judging a change by its most impressive example.
You can start with a simple worksheet. What matters is that each row represents a meaningful task, carries the context needed to judge the result and records what an acceptable outcome looks like. The first version should help the team find disagreements and failures, not create a reassuring score without explanation.
For a large language model, or LLM, evaluation should connect the generated response with the application's purpose. A convincing sentence is only one part of a completed task.
The example throughout this article is a fictional internal assistant that answers product-policy questions using approved documents. It illustrates evaluation design rather than a particular vendor, model or customer result. The same approach can be adapted to extraction, classification and other AI features, with different criteria for each task.
Keep in mind
Key takeaways
- Build each case around a real task, its context and an observable expected outcome.
- Separate retrieval, answer quality and permission failures when reviewing results.
- Keep reproducible cases and record the decision that follows each evaluation.
Define the decision the evaluation will support
Begin with a change you need to assess. Perhaps the team is replacing a retrieval method, revising a prompt or adding another document source. Write the decision in plain language: should this change proceed, be revised or remain an experiment? That statement determines which tasks and failure cases belong in the evaluation.
A general goal such as “improve answer quality” leaves too much open. For the policy assistant, a more useful question is whether the new retrieval setup finds the applicable policy and allows the answer to reflect its exceptions. That separates retrieval from tone, formatting and other qualities that may matter but do not answer the same question.
Name who will make the decision and who can judge the work. A product owner may understand the intended user experience, while a policy specialist can assess applicability. Engineering can explain execution failures, but should not have to invent the business meaning of a correct answer. Identify the required perspectives before collecting a large pile of examples.
Also state what this evaluation cannot establish. A small, deliberately selected dataset can expose problems in the tasks it covers. It is not automatically a representative estimate of all live traffic, and a successful run does not establish security, accessibility or operational readiness. Keeping those boundaries visible makes the findings easier to use responsibly.
Build the task inventory before writing prompts
List the things users need to accomplish, using the language of their work. For our assistant, tasks might include finding an upgrade condition, comparing two policy versions and explaining which information is needed to answer an eligibility question. Start with these activities rather than a list of prompt-writing tricks.
Collect examples from legitimate sources available to the project: approved research notes, support workflows, product documentation or properly governed operational records. Record where an example came from and whether it has been edited. Access and retention requirements should be agreed with the people responsible for the material. Do not move confidential conversations into a shared evaluation sheet simply because they are useful examples.
Preserve the facts that make the task difficult. Removing a date, document conflict or product variant may make an example easier to share, but it can also remove the behaviour you intended to test. If the original cannot be used, construct an illustrative replacement with the same decision problem and mark it as synthetic.
Group the inventory by task and by important conditions. You might distinguish a direct fact lookup from a question requiring several passages. Note when information is missing, outdated, restricted or contradictory. These labels help the team see the coverage it has chosen and the gaps it has not yet addressed.
Give every row enough context to be judged
A prompt alone is usually an incomplete evaluation record. The same question can require different answers depending on the user's permissions, the document collection and the date of the applicable policy. Include the inputs and conditions that affect the expected result.
For the fictional policy assistant, each row can identify a task, a question, an approved source version and the facts the answer must preserve. Add a criterion describing how to handle missing information. A question without an answer in the available documents should not be silently relabelled as a failed lookup when the intended behaviour is to ask for clarification.
Scroll to view all columns →
| Field | Example purpose | Review question |
|---|---|---|
| Case ID | Stable reference for a failure | Can we find the same case next week? |
| Task and input | The user's actual request | Is the task understandable without guessing? |
| Context version | Documents and permissions used | Did both runs use the same conditions? |
| Expected behaviour | Required facts or next step | What would make the result acceptable? |
| Observed result | Output and relevant execution evidence | What actually happened? |
| Review outcome | Decision and reason | Can another reviewer follow the judgment? |
Keep the record practical. If a field never affects a judgment or helps reproduce a result, ask whether it is worth maintaining. Conversely, when two reviewers disagree because the applicable policy version is absent, add that context rather than resolving the dispute by preference. The worksheet should improve as the evaluation exposes what matters.
Write criteria that different reviewers can apply
A criterion such as “the answer is good” hides several decisions. Break it into observable requirements. In our example, the response must use the applicable policy, preserve its qualifying condition and avoid inventing an account fact. The response can be concise and readable while still failing any of those checks.
Describe what earns each outcome. A simple pass, revise or reject decision may be enough for an early review. If you use a numerical scale, define its levels with examples. A score becomes more useful when the team can explain why a result belongs at one level rather than another. Extra decimal places do not resolve an unclear rubric.
Run a small calibration exercise before reviewing the whole dataset. Give the same cases to two people, compare their judgments and discuss the disagreements. Sometimes the model result is ambiguous; sometimes the evaluation instruction is. Update the criterion or the expected context where necessary, keeping a record of the decision.
Google Cloud's evaluation documentation describes datasets and several forms of evaluation criteria, including rubrics and computation-based methods. The worksheet in this article is our editorial approach to preparing useful review material. It is not a Google scoring standard, and it does not require adopting a particular evaluation service.
Reference: Gen AI evaluation service overview ↗
Use reference answers without treating wording as truth
A reference answer can help a reviewer understand what the case requires. It can also become misleading when the team treats one phrasing as the only acceptable output. For a policy question, several responses may communicate the same supported facts and appropriate next step.
Separate required content from optional presentation. Record the eligibility condition, the applicable product version and the need to request a missing contract date. Then allow reasonable variation in wording. Exact string matching is useful for some outputs, such as a required identifier or a constrained field, but it is a poor substitute for judging the meaning of an open-ended explanation.
Inspect the reference material itself. Who prepared it, which source supports it and what assumptions does it contain? A generated reference answer should be reviewed before becoming the basis for judging another model. Otherwise the evaluation can reward the same unsupported claim twice and make the process look more rigorous than it is.
Include cases with no complete answer. Specify whether the system should ask a question, return a partial result or decline the task within the available context. Those cases reveal whether the product can handle its limits. They also stop the dataset from teaching the team that every user request must end in a confident paragraph.
Separate retrieval failures from answer failures
In a retrieval-augmented generation system, or RAG system, the answer depends partly on which material reaches the model. When a result is wrong, inspect that material before changing the prompt. The relevant passage may have been absent, outdated or mixed with a different product's policy.
For each case that needs document evidence, identify the passage or information that ought to be available. Record what was actually retrieved and whether it was accessible to the user. Then judge the answer in that context. This helps distinguish a missed source from a response that misused a source it had already received.
The distinction affects the next engineering step. If retrieval never finds the qualifying exception, a new tone instruction will not supply it. If the correct passage is present but the response ignores it, improving search alone may not solve the problem. Keep both observations in the case record so later changes can be assessed against the same failure.
Avoid letting a correct final sentence erase an unreliable route to it. The assistant may have arrived at the right conclusion using a source that does not support the claim. That matters when the design promises inspectable evidence. Include a criterion for the relationship between claims and sources when that relationship is part of the product's purpose.
Related reading: Design AI answers with inspectable sources
Add the cases that could change your decision
An evaluation collection filled with straightforward successes is easy to maintain and easy to misread. Ask which plausible failure would make the team pause the change. For the policy assistant, examples include contradictory document versions, a question about an unsupported product and a request requiring an account detail the system does not have.
Include variations in the form of the input. A user may paste a long message, misspell a product name or ask two related questions at once. Choose variations grounded in the intended use rather than assembling a random list of strange prompts. Each case should have a reason to exist and an outcome the reviewers can assess.
Test relevant boundaries of permission and scope. A user should not gain access to another team's restricted policy through a generated summary. A document containing instructions aimed at the assistant should not automatically become authority for an unrelated action. Agree these cases with the people responsible for the application and its security controls.
Label the coverage deliberately. A difficult-case set is useful because it concentrates on meaningful problems; it should not be described as a neutral sample of everyday usage. Keep that collection distinct from any sample intended to estimate live task frequency. Both can inform a decision, but they answer different questions.
Change one important condition at a time
When comparing two implementations, record what changed and what stayed fixed. A retrieval experiment becomes hard to interpret if the prompt, document set, model and rubric also change. Sometimes several changes must travel together, but the result should then be described as a comparison of packages rather than proof about one isolated component.
Preserve the original outputs and the conditions of each run. Include the relevant configuration, source versions and evaluation instructions. The right level of detail depends on the system, but another person should be able to understand which setup produced a result. A spreadsheet containing only the latest score cannot provide that history.
Review case-level movement as well as any summary. A new version may solve several routine lookups while failing one important permission case. The average can improve even when the change introduces a reason to delay release. Decide in advance which kinds of failures need separate treatment, then show them prominently in the review.
Allow for variability when the behaviour being tested can vary across runs. Repeating selected cases may reveal that a seemingly solved problem is inconsistent. Record the repetitions rather than choosing the best output. The aim is to understand the observed behaviour well enough to make a decision, not to produce the most flattering demonstration.
Use automated judging as a tool within the review
Some checks can be automated directly: required fields exist, an identifier matches the expected record or the output follows a specified structure. Other checks involve interpretation. A model-based judge may help organize those reviews, but its judgment needs its own scrutiny.
Try the proposed judging method on cases that knowledgeable people have reviewed. Inspect where it disagrees, including cases with confident wording and weak evidence. Check whether it rewards a verbose answer, follows irrelevant instructions in the evaluated text or misses a condition important to the business task. Keep the judge's instructions and version with the evaluation record.
Separate the judge from the acceptance decision. An automated result can flag material for review or support a defined criterion; it should not quietly turn an unresolved policy question into a pass. Give reviewers access to the input, output and source context needed to understand the flag. Where the task cannot be judged from the available material, record that limitation.
Choose the amount of automation according to the review problem. A small early dataset may benefit most from careful manual judgment and clearer examples. As the collection grows, automate the checks whose meaning is already understood. Building a more elaborate scoring pipeline will not settle a disagreement about what a successful task should accomplish.
Read one failed case from input to decision
Consider an illustrative row for the policy assistant: a user asks whether their account can use a scheduled export. The approved document says that access depends on account type, and the supplied context does not identify that type. The required behaviour is to preserve the condition and request the missing detail.
A response saying that scheduled export is available fails that criterion, even if the feature exists and the prose reads well. A response saying the feature is never available also fails. A useful result can explain the documented condition and ask which account type applies. The rubric should allow different wording while preserving that decision.
Now inspect the retrieval record. If the applicable passage was absent, record the retrieval problem. If it was present and the answer ignored it, record that observation separately. When both occur in different runs, keep both records. The purpose of the case is to understand behaviour, not force every failure into one convenient category.
A reviewer might disagree because they assume the assistant can access an account database. That disagreement identifies missing evaluation context. State whether the integration exists, whether this user is authorized to use it and what information it returns in this case. Do not let reviewers fill those gaps differently and then average their judgments.
Finally, record the delivery decision. The team may revise retrieval, clarify the task instruction or change the interface to request the missing detail. Repeat the affected case and the cases that could regress because of the change. The row now carries an input, a criterion, an observed failure and a reason for the next step, which is far more useful than a low score without explanation.
Keep failures without turning them into rehearsed examples
When a case exposes an important problem, keep it as a regression check. Preserve the reason it failed and the change intended to address it. This lets the team see whether a later model, prompt or integration update reintroduces a known issue.
Also keep some evaluation material separate from the examples used to develop the system. If every failure is copied into the prompt or used to adjust the implementation, success on those same rows becomes less informative about unseen tasks. The practical separation depends on your process, but the distinction between development examples and independent review cases should be explicit.
Refresh the collection when the product changes. New policies, different user groups and additional tool access can make an old dataset incomplete. Retain version history so a change in coverage is not mistaken for an improvement in behaviour. Retire obsolete cases with an explanation rather than quietly removing difficult rows from the report.
End each evaluation with a decision record: what was tested, which conditions were covered, what failed, what remains unknown and what happens next. If an external development company is involved, ask who owns that record and the reusable dataset after handover. Useful evaluation leaves the next team with a clearer way to judge the product, not just a score from the final demo.
Related reading: Plan the AI pilot handover · Compare AI development companies
Read the original thinking
Sources & further reading
This is original Point of AI analysis. The sources support the statements attributed to them. Hypothetical examples and suggested questions are editorial guidance.
- Gen AI evaluation service overview ↗Google Cloud · Checked November 13, 2026
Supports task-specific datasets, explicit evaluation criteria and inspection of individual results. The starter worksheet and suggested review process are editorial recommendations, not a Google certification or release standard.