Did the Course Work? Measuring Something Other Than Satisfaction

Most course evaluation measures satisfaction. It is easy to collect, and it correlates only loosely with whether anyone learned anything.

Learners can enjoy a course that taught them little, and can rate poorly a demanding course that changed what they can do. Neither is unusual.

Why satisfaction misleads

• It measures the experience, not the outcome.
• Difficulty lowers ratings even when it is the difficulty that produced the learning.
• It is collected at the end, before anyone has tried to use what they learned.
• Only some people respond, and they are not a random sample — the delighted and the annoyed both respond more than the middle.

Still worth collecting — dissatisfaction points at real problems. Just do not treat it as evidence of effectiveness.

What to measure instead

• Completion, by section. Where people stop tells you where the course fails.
• Exercise completion — the strongest available proxy for actual learning.
• Assessment results by topic — a topic everyone gets wrong points to a lesson, not to the learners.
• What they can do afterwards — the real measure, and the hardest to collect.
• Whether they used it — ask a few weeks later, which is when the answer becomes meaningful.
• Whether they recommend it to someone with the same problem — a stronger signal than a rating.

Collecting evidence you can act on

• Ask before and after. A short self-assessment at the start and end shows change rather than absolute level.
• Ask what they can now do that they could not before — an open question that produces the most useful answers you will get.
• Follow up after a month with two questions: did you use it, and what was missing.
• Collect examples of work produced during the course, with permission — this is direct evidence rather than opinion.
• Ask people who did not finish why. This is where the most actionable information lives, and almost nobody collects it.

Asking better questions

• Ask about specifics, not overall quality — which lesson was clearest, which was hardest to follow.
• Ask what was missing rather than what was good.
• Ask what they would tell someone considering it — people answer this more honestly than a direct rating.
• Avoid leading questions. Asking how much someone enjoyed a lesson produces a different answer from asking what they thought of it.
• Keep it short. Three good questions get answered; fifteen do not.

Reading the data honestly

• Look for patterns, not individual comments. One strong complaint is not a trend; three similar ones are.
• Weight the criticism — it contains more information than the praise.
• Check who responded before generalising.
• Separate what you can change from what you cannot. Some feedback is about the learner's circumstances rather than your course.
• Be careful with small numbers — twelve responses do not support confident conclusions.

Acting on it

• Fix the biggest drop-off point first — it affects the most people.
• Change one thing at a time so you can tell what worked.
• Give a change enough time before judging it.
• Tell learners what you changed because of their feedback. It costs nothing and it substantially increases future response rates.
• Keep a record of what you changed and why, so you do not undo it later.

Being honest about limits

• You cannot control what happens after the course — whether someone applies it depends on their job, their time and their circumstances.
• Do not claim outcomes you have not measured. If you have not tracked what learners achieved, do not put it on the sales page.
• Small-scale evidence is still evidence, but describe it accurately — a handful of examples is a handful of examples, not a result.
• Improvement is incremental. A course gets meaningfully better over years, not over one revision.

The simplest useful measure

If you do only one thing:

Ask everyone who finishes, a month later: what have you done with it?

The answers tell you what the course actually produced, which parts stuck, and what people needed that they did not get. It takes one email and it is worth more than any satisfaction score.

One thing worth remembering

A course that people enjoy is pleasant. A course that changes what people can do is the thing you set out to build.

Measuring only the first is comfortable — and it is why many courses stay the same for years while feeling successful.

FAQ