Case study 04 · Evaluation & quality assurance
A practical way to tell the difference between a programme that was delivered and one where students actually learned.
Completion rates and happy-face surveys are nice, but they don’t show whether students can reason about AI, check its answers or use their skills somewhere new. Leaders need clear signals that link teaching quality, learning progress, risks and next steps.
The scorecard groups measures into four areas: how students reason, curriculum quality, teacher readiness and how reliably the programme runs. Each measure has a definition, a data source, how often it’s reviewed, who owns it and a decision rule, so collecting data actually leads to action.
| Dimension | Example indicator | Method / caveat |
|---|---|---|
| Learner reasoning | % meeting “Secure” on verification rubric | Assess with common tasks; report sample size |
| Learning progression | Change in rubric level across checkpoints | Same criteria; avoid claims without comparability |
| Content quality | % of lessons passing age/accuracy/safety QA | Use documented review checklist |
| Educator readiness | % demonstrating skill in scenario-based assessment | Observation or scenario task, not confidence alone |
| Delivery reliability | On-time delivery and critical issue resolution | Track issue severity and exclusions |
| Feedback to action | % improvement actions closed by due date | Link actions to owner and evidence |
What we notice: Students hand in polished AI answers but hardly ever cite sources. Why it might happen: the assessment rewards the final product; checking sources isn’t modelled consistently; prompts don’t ask for evidence. What we try: add a check-and-cite routine, update the rubric and review a sample of answers at the next checkpoint.
This is a made-up example to show the design, not a description of any real employer programme.
| Action | Owner | Evidence required | State |
|---|---|---|---|
| Introduce a check-and-cite routine | Curriculum lead | Sample of responses at next checkpoint | In progress |
| Revise the verification rubric | Assessment lead | Updated rubric, moderation notes | Done |
| Run a moderation session | Programme manager | Agreement across reviewers | Not started |
| Measure | Synthetic value | Definition |
|---|---|---|
| Learner reasoning | 58% | Meeting “Secure” on the verification rubric |
| Curriculum quality | 91% | Lessons passing age, accuracy and safety review |
| Educator readiness | 72% | Demonstrating skill in a scenario task |
| Delivery reliability | 94% | Sessions delivered on time |
| Verification rubric, Checkpoint 1 | 31% | Share of learners meeting “Secure” |
| Verification rubric, Checkpoint 2 | 38% | Share of learners meeting “Secure” |
| Verification rubric, Checkpoint 3 | 49% | Share of learners meeting “Secure” |
| Verification rubric, Checkpoint 4 | 58% | Share of learners meeting “Secure” |
