Worked Examples

To illustrate the Three-Step Process in practice, we provide two distinct worked examples at different stages of product development:

• A citizen-facing chatbot before launch, where evaluation supports a decision on deployment readiness.
• A case-decision assistant at prototype stage, where evaluation helps identify areas for improvement and prioritise fixes.

Both follow the same underlying process, but how "good" differs depends on the product's mission, evaluation purpose and stakeholders involved.

---

Case Study 1: A citizen-facing chatbot before launch

The situation

You have been working with a government agency to develop a public-facing chatbot answers citizens' questions about government schemes and services. It has already been tested during the Proof-of-Value (POV) phase, and is about to go into controlled production, and you, the product owner, need to decide whether the chatbot is ready.

The three-step process

1Reflect on purpose

Why evaluate?

Support a decision: launch or hold for further improvements.

What is it for?

To provide citizens with correct and helpful answers on government schemes and services in a friendly, accessible manner.

↓

3Agree the list

Agreed withthe product team, relevant agency heads and senior management in charge of the chatbot's deployment.

2Expand for coverage

DimensionGranularityPerspectiveAccurateReplies are grounded in the available information as the source truth.FunctionalityEnd-to-endSystemRelevantReplies address the question asked directly.FunctionalityEnd-to-endSystemComprehensiveAll common details of schemes and services are covered in the response.FunctionalityEnd-to-endSystemDeclines out-of-scope queriesThe chatbot only answers questions on government schemes and services, and nothing else.FunctionalitySafetyEnd-to-endSystemFriendlyReplies are delivered in a friendly and approachable manner regardless of audience.FunctionalityEnd-to-endBusinessConsistencyRepeated replies concerning the same question of the same topic provide similar depth in information and scope.FunctionalityReliabilityEnd-to-endSystemResponds quicklyReplies arrive quickly enough that citizens do not give up on the conversation.ReliabilityEnd-to-endOperational

…

Some reflections

• The launch decision shaped the list. Because the chatbot is about to go into production, the choice of specifications follows from what a launch decision needs:
  • Mostly end-to-end, because the decision turns on what citizens actually experience, not on the performance of any individual components that make up the system.
  • Safety, because a live bot that improvises beyond its remit is a real launch risk, so declining out-of-scope queries is treated as a safety requirement.
  • Reliability, because it will soon serve the public repeatedly and at scale, so holding consistent depth and scope across similar questions is a reliability requirement.
• An early draft might have started with "replies are of good quality". Refining "quality" with the stakeholders turned it into several specifications: accurate, relevant and comprehensive, friendly, consistent.
• The team could also add "replies are succinct" to the list. The senior management in this case study did not consider succinctness critical, but another product team could reasonably keep it. There is no right or wrong choice here, only what matters to your stakeholders.

Case Study 2: A case-decision assistant at prototype stage

The situation

You, an application developer, have been prototyping an assistant that supports officers handling appeals for a government grant. It digests the documents the applicant submitted, checks them against rules held in several systems, and recommends supporting or declining the appeal, with a rationale generated through a summary of the appeal.

The three-step process

1Reflect on purpose

Why evaluate?

Improve the system: identify current failure modes and whether current features are working as expected.

What is it for?

To support officers in making informed, grounded and accurate decisions on handling appeals.

↓

3Agree the list

Agreed withthe product team, senior case officers, senior management that decides on the product roadmap.

2Expand for coverage

DimensionGranularityPerspectiveSummary is accurateEvery claim within the summary is supported by the submitted documents.FunctionalityComponentSystemSummary covers all key informationThe summary includes all necessary information from the submitted documents.FunctionalityComponentSystemCorrect tool callsThe system invokes necessary the correct tools to parse the submitted documents and check against the rules in different systems.FunctionalitySubsystemSystemCorrect recommendationsProposed recommendation follows the golden set that is defined by human annotators.FunctionalityEnd-to-endSystemRationale reflects the recommendationThe provided rationale reflects the recommendation provided by the system.FunctionalityEnd-to-endSystem

…

Some reflections

• The improvement goal shaped the list. Because evaluation is here to surface and prioritise failure modes, the choice of specifications follows from what diagnosis needs:
  • Finer granularity, because the decision is where to fix, so each stage gets its own specification (summary accuracy and coverage at the component level, tool calls at the subsystem level) rather than a single end-to-end verdict.
  • Stage-level traceability, because with those per-stage specifications a wrong recommendation can be traced to the stage that caused it, whereas an end-to-end specification alone cannot show which stage failed.
• Response time is deliberately missing. Low latency is generally "good", but officers do not process an appeal the moment it arrives, so speed is not critical at this stage. This is why product context matters.
• The team could also shape the rationale specification differently. One stakeholder may want rationales that are comprehensive (every applicable rule cited), another may want them precise (only the decisive reason stated). Understand what your stakeholders value at this stage, not after the results arrive.

---

Overall thoughts

From both examples, we can see that the definition of "good" is highly context-dependent. Each one reflects the unique challenges and priorities of its respective use case.

Hence, the goal is not to produce the longest list, but one that meaningfully captures what matters for your product, to you and your relevant stakeholders.

Before you head off

Not satisfied with your list yet? Revisit the Three-Step Process to refine your definition of "good" and ensure it aligns with your evaluation purpose and stakeholder expectations.

Otherwise, head over to 2. Design to begin your evaluation design.