Discovery and simulation
Map the pathway, hazards, users and measures. Test prototypes with synthetic or consented non-live examples.
Evidence and controlled pilots
Demonstrate safety, usefulness and value in the real pathway.
The evaluation plan must test what happens from capture through review and action—not merely whether people can upload an image.
Pilot sequence
Four gated phases
Shadow evaluation
Run image capture and comparison without allowing output to direct care; compare with normal assessment.
Limited live pilot
Use a narrow cohort, trained team, daily oversight, conservative escalation and predefined stopping rules.
Scaled service
Expand only after approved evidence, resolved hazards, operational capacity and ongoing surveillance are in place.
Evaluation questions
What must the evidence answer?
Can people capture useful images?
Measure first-time acceptance, retake rate, comparability, device variation, support need and differences between user groups.
Does the workflow support timely review?
Measure submission-to-open, open-to-decision, overdue reviews, failed contact, appointments and escalation completion.
Does it improve safety?
Look for missed or delayed deterioration, false reassurance, inappropriate escalation, wrong-person events and incident severity.
Is it clinically useful?
Ask whether the timeline changed confidence, information quality, prioritisation or management when assessed in full context.
Is it equitable?
Analyse uptake, completion and outcomes by relevant protected characteristics, deprivation, digital access, language and disability.
Does it create value?
Measure staff time, avoided and added contacts, travel, appointments, treatment changes, infrastructure and support cost.
Core measures
A balanced pilot scorecard
| Domain | Example measures | Guardrail |
|---|---|---|
| Safety | Serious incidents, missed escalation, delayed review, false reassurance, urgent-route use. | Immediate review and predefined pause criteria. |
| Performance | Image quality, alignment success, disagreement with reviewers, uncertainty and subgroup performance. | No single metric substitutes for clinical judgement. |
| Operations | Queue volume, response time, backlog, contact attempts, closed loops and downtime. | Capacity threshold and escalation when breached. |
| Experience | Ease, trust, comprehension, accessibility, burden and willingness to continue. | Alternative non-digital route remains available. |
| Equity | Access and outcomes by demographic, language, disability, device and connectivity factors. | Investigate disparity before scale-up. |
| Economics | Implementation, support, clinician time, travel, appointments, admissions and treatment consequences. | Report costs displaced as well as costs avoided. |
Study governance
Define the rules before looking at results
- Approved protocol, cohort, endpoints, analysis plan and data-management plan.
- Clear distinction between service evaluation, audit, research and regulatory clinical investigation.
- Appropriate consent or other lawful basis, ethics and information-governance review.
- Independent or appropriately separated safety oversight and conflict-of-interest declarations.
- Version control for the software, model, instructions and clinical pathway throughout the study.
- Publication of favourable, neutral and unfavourable findings with limitations.
Start with a narrow, measurable pathway
Use the checklist to turn this evaluation framework into a local pilot pack with named ownership and stopping rules.