Measuring outcomes

Every program evaluation eventually arrives at the same question: did it work? Participant self-report is the most common way to answer it, but the design determines the credibility and trustworthiness of the resulting information. Three designs dominate practice: global ratings of change at program end, traditional pre-post, and retrospective pre-post. The first is the most vulnerable to bias. Both pre-post designs are more rigorous; the choice depends on what is being measured. 

Global change 

Many organizations assess program outcomes with self-report questionnaires administered at the end of participation, where respondents rate statements such as "I have a better understanding of healthy relationships as a result of this program" and "This program has helped me feel less stressed" on a 5-point scale. These items are easy to administer, simple to aggregate for reporting, and typically produce favorable results. 

Unfortunately, this design also tends to artificially inflate program outcomes due to several forms of bias. First, participants may inflate the value of the program to justify the time, effort, and opportunity cost their participation represents (“effort justification”).¹ Second, the item asks respondents to recall where they stood before the program, compare that to where they are now, and judge how much of the difference the program caused. Memory of a past state is rebuilt rather than retrieved – if they believe a program should have helped, they remember starting out worse than they did (recall bias).² Third, participants who are grateful for services, have formed connections with staff, or feel indebted to the organization may feel pressure to respond positively (social desirability bias).³ Fourth, the item framing adds to this pressure: a statement that credits the program signals the expected answer (a “demand characteristic”).⁴ 

We don’t recommend global change ratings at program end for assessing outcomes, though they are fine for gauging satisfaction or perceived helpfulness. 

Traditional pre-post 

For outcomes that don’t require special knowledge imparted by the program itself to accurately rate –such as attitudes, feelings, and behaviors – the best option is the pre-post design. In this design, respondents rate identical items (e.g., How often have you felt stressed in the past two weeks?) at least twice: once before the intervention begins (the "pre") and once after it ends (the "post"). Evaluators calculate the change for each respondent and average those changes across the cohort.  

The upside of this format is that it produces more accurate outcome estimates for these outcomes by reducing or eliminating effort justification, recall bias, social desirability bias, and demand characteristics. The downside is that it requires tracking program participants over time, and that it can mask a specific class of outcomes, covered next. 

Retrospective pre-post 

For outcomes that do require special program-imparted knowledge to accurately rate – such as changes in program-specific knowledge and skill – we recommend the retrospective pre-post. At baseline, respondents often do not know what they do not know, so they are vulnerable to overestimating their “pre” knowledge or skill. While their post ratings may be accurate, their potentially inflated pre ratings can depress change scores and mask program outcomes. This phenomenon is known as response shift bias.⁵  

In the retrospective pre-post design, respondents rate their “pre” and “post” knowledge or skill after the program is done. Both ratings come from the same informed perspective, so the comparison holds, and change scores are calculated as in a traditional pre-post design. 

While this design re-introduces some recall and social desirability bias, the trade-off in better informed judgments is worth that cost. This format also offers a practical option when collecting baseline data is not feasible (e.g., evaluations that begin after a program is underway).  

How to decide between traditional and retrospective pre-post 

Design	Use when participants…	Outcome types	Example item
Traditional pre-post	Can accurately rate where they stand before the program	Attitudes, feelings, behaviors	How stressed have you felt in the past two weeks?
Retrospective pre-post	Can’t judge where they started until the program does its work	Program-specific knowledge, skills	I understand the signs of suicide risk

 The common thread: ask people where they stand, not how much they've changed, and let the comparison do the work. The only real decision is when to ask about the starting point: before the program, or after it has taught participants what to look for. Neither design proves the program caused the change; that takes a comparison group. But both give far more credible evidence of change than asking participants to grade the program's effect on them. 

---

¹ Aronson, E., & Mills, J. (1959). The effect of severity of initiation on liking for a group. Journal of Abnormal and Social Psychology, 59(2), 177–181. https://doi.org/10.1037/h0047195

 ² Conway, M., & Ross, M. (1984). Getting what you want by revising what you had. Journal of Personality and Social Psychology, 47(4), 738–748. https://doi.org/10.1037/0022-3514.47.4.738

³ Geldhof, G. J., Warner, D. A., Finders, J. K., Thogmartin, A. A., Clark, A., & Longway, K. A. (2018). Revisiting the utility of retrospective pre-post designs: The need for mixed-method pilot data. Evaluation and Program Planning, 70, 83–89. https://doi.org/10.1016/j.evalprogplan.2018.05.002

⁴ Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903.

⁵ Howard, G. S., & Dailey, P. R. (1979). Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64(2), 144–150.

---

Liz DiLuzio is an evaluator at the Behavioral Health Improvement Institute (BHII) at Keene State College.