Design: Your Evaluation

Recap: what you bring into this chapter

You should have the following:

• A concise, stakeholder-agreed list of the specifications your product must show (from Define: What "Good" Looks Like)

If you don't have any of the above, complete those chapters first and return here after.

You have now consolidated and solidified what "good" looks like for your product into a concise, agreed list. The next step is to develop an evaluation design that sets out how to assess your system's performance on each specification in a structured way.

---

What makes up the evaluation design

We can view the diagram below as a framework for your evaluation design. It has two main components:

2b · Evaluation Set DesignWhat you run against the system

System InputWhat is ran

SystemProduct under test

Subject(s) of JudgementWhat is returned to be judged

2a · Judgement DesignHow you measure

ArtefactWhat is judged

Judge

⋯metric, criteria

VerdictDoes the system meet what 'good' looks like?

---

How this chapter is organised

For the following two sub-chapters, we will approach the evaluation design in reverse: first deciding how to measure, followed by what to measure on.

This order makes the design process easier to follow. By first establishing how performance will be judged, we can then design an evaluation set with the right test cases and supporting information to enable those judgements.

1. 2aJudgement DesignHow do you measure each specification?
2. 2bEvaluation Set DesignWhat do you run that measurement over?

What you will have by the end

• A judgement design (2a)For each specification: the artefacts to judge, the judges to use and the judgements they return, summarised into metrics with acceptance criteria
• An evaluation set (2b)The test cases you run that judgement design over, with the supporting information each judgement needs