Jeffrey Saltzman's Blog

Enhancing Organizational Performance

Results not Typical

leave a comment »

[tweetmeme source=”jeffreysaltzman”]

“I got this body while eating pizza, hamburgers”, and she then leans forward whispering into the camera “even chocolate!” Before and after pictures are displayed on the screen. While the shapely product promoter is telling you how she lost all that weight and became irresistible by eating the offered foods there is a line on the right-hand bottom corner of the screen, “Results not typical”.  I wonder what result was not typical. That people who use this product end up on TV promoting it? Or that people who use the product actually lose weight? Or that people who eat the advertised food actually think it is any good? Maybe “Results not typical” refers to all three. One has to wonder given the schlock nature of the ad just what warning their fine print is conveying.

Of course, sadly, the ad would not be running, and it has been around awhile, if it did not work in attracting people to use the product. We should all be aware of the weakness of the case study approach as well as approaches that claim success without appropriate experimental design. How would you evaluate the above statement if you treated her claim “eat this food and lose weight” as an organizational program or initiative and wanted to determine if in fact you could place any stock in those claims?

Program evaluation suffers a history of skepticism often due a history of poorly conceived evaluation methodologies. One of the most widely used designs for program evaluation is one in which 1). a single group is given a baseline measure, then 2.) the program is implemented and then 3.) a post implementation measurement is made to determine the effect of the program. This approach is fraught with problems. Let me illustrate. Let’s use the following statement as an example:

  • A school system wants to know if the investment it is making in advanced teacher training is improving educational attainment among its students.

In this example a baseline measure regarding standardized student achievement test scores is collected prior to the implementation of the new teacher training program. In addition surveys can be done of the students asking about their comfort and mastery with the subjects covered by the teacher training program. After the teachers receive their training and the next class of students comes through, the measurement process gets repeated. Students on average now report that they feel more comfortable with the targeted subjects and test score in fact are moving up. The teacher training program is determined to be a success and additional funds are poured into teacher training. What was not taken into consideration using this approach is the fact that the students got new textbooks with vastly improved course material and the class schedule was redesigned so that the students spent more time on the targeted subjects. What was thought to be an outcome of teacher training was actually better course material and more time spent devoted to the subject.

There are various approaches that could improve the ability of the school system to improve its evaluation. Here is one. If the above field experiment had been carried out as follows the results would have been much clearer. In this alternative approach the same setup is used, but only half the teachers in the first round receive the training. When the next round of students come through the program the students whose teachers received the training are compared to those whose teachers did not (a control group), holding everything else constant. Holding everything else constant in this case means that all students received the new texts and all students had their schedules changed. Now when we compare the students whose teachers received the training to those who did not, we might find that all student scores improved but the students whose teachers received the advanced training had even higher scores and felt even more comfortable regarding their mastery of the subject than those students whose teachers did not. In this case a more confident determination can be made that the advanced teacher training did help to improve test scores.

The criticism of this approach is that it is not right to withhold a potentially beneficial experience from those in the control group. And the answer to that is that in certain circumstance it is not right to withhold that potential benefit (experimental drugs being used in life threatening circumstances is one case). However in the above example it is truly unclear whether there is any benefit to the students whose teachers went through the advanced training vs. those who did not. And their certainly would be a long-term benefit to the school system by knowing the true impact of that training experience.

© 2010 by Jeffrey M. Saltzman. All rights reserved.

Visit OV:

Written by Jeffrey M. Saltzman

October 20, 2009 at 10:20 am

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: