Jeffrey Saltzman's Blog

Enhancing Organizational Performance

Measuring what you are Managing – Part 1

leave a comment »

[tweetmeme source=”jeffreysaltzman”]

There are times when rigorous measurement of programs, processes or choices can greatly aid in the decision making process and surveys can be used in that evaluation. For instance, in organizations when the amount of resources that can be brought to bear are limited and difficult decisions need to be made about the deployment of those resources, a measurement process that gives insight into the expected benefits of action A vs. action B can be especially helpful. Consider the following situations:

  • A school system wants to know if the investment it is making in advanced teacher training is improving educational attainment among its students;
  • The GAO wants to know if efforts in information dissemination by government agencies can be documented as furthering government agency goals;
  • A corporation wants to know if changes made in a reorganization are achieving the desired impact in terms of increasing organizational effectiveness and customer satisfaction;
  • A zoo wants to know if it should proceed with increasing elementary school outreach by investing in an animal travel program, developing an on-site hands-on children’s zoo, or investing in bringing in more exotic animals and building new exhibition space in order to maximize attendance;
  • A retailer wants to know if it should invest in more physical retail locations, beef up its website or send out additional catalogs with its limited budget dollars;
  • A benefits departments wants to know how the company’s redesigned benefits package is being perceived by employees and how well the execution of the benefits program is being conducted by the new outsourced provider;
  • A refiner wants to know if it should invest in a new Greenfield plant (a tangible asset) or if there would be greater ROI if it invested the equivalent dollars in additional training for all of its workers that would be expected to increase efficiency and throughput in its existing manufacturing assets (an intangible asset);
  • A sales department wants to measure its sales funnel of potential deals, to determine if its marketing and sales efforts are having the payback it desires and to predict future sales volumes for the organization.

All of these decisions or potential decisions can be enhanced by an effective effort aimed at program evaluation. Clearly having effective measures is invaluable to those trying to manage these kinds of decisions. Where to invest, how much to invest, when to invest are critical decisions and managers would be well served to seek out systematic measures to enhance their judgment. The American Evaluation Association defines evaluation as “assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness.” While organizations vary in their efforts at program evaluation one interesting statistic comes from NASA-Goddard which has determined that for them between 7% and 9% of a total program’s budget should be spent on evaluation, and a recent Air Force initiative devoted 15% of the project budget to a rigorous evaluation of whether their new inventory control system was working as planned. Program evaluation though has been around long enough and has been practiced by a wide enough variety of people that not only have myths sprung up about program evaluation but other documents that attempt to debunk the myths of program evaluation are also in existence.

Traditionally when organizations have thought of program, policy or initiative evaluation what springs to mind is a list somewhat like the following as areas to be evaluated:

  1. Perception of Benefits –medical, life, dental, disability, wellness, stock options, matching contribution to retirement, vacation, maternity policy
  2. Views of Pay & Bonuses
  3. Health Safety and Environment (HSE) Emphasis and Policy,
  4. Effectiveness of Diversity Programs,
  5. Work Life Balance Initiatives,
  6. Psychological Recognition Efforts,
  7. Development & Training Opportunities,
  8. Advancement Systems,
  9. Physical Conditions,
  10. 10.  Job Security.

One of my goals is to broaden out that traditional definition to include a broader array of various types of programs, policies and initiatives that an organization may undertake and would benefit from a rigorous evaluation methodology.

Many program evaluations have been sneered at as wasteful efforts due to poor evaluation methodologies being applied, but another factor that creates the typical cynicism is that whether a program gets continued funding may be somewhat contingent on issues outside of the effectiveness or the evaluation of the program. Common problems facing program evaluation consist of unclear program goals (so how do you know if you have accomplished the goal?), difficulty in collecting data, a lack of cooperation among those responsible for implementation (those whose plates are already full) and the politics of and vested interests surrounding various programs.  Yet with a methodical approach insights can in fact be gained without too much pain into whether the program is having the hoped for impact.

Program evaluation can take place within a specific window in time or it can be an ongoing activity. For instance, an organization might implement a phased-in approach to a new program, or a new product/service etc. and decide that after 6 months the program will be evaluated to determine its impact, and whether funding should be continued at the current level, increased or diminished. If there is a positive finding to the evaluation effort, the program would be rolled out to the rest of the organization. Other organizations may not use a phased in approach and may simply have two or more programs being performed simultaneously in the organization and then conclude possibly, that it cannot afford all of them, so which ones are benefiting the organization more? An evaluation might be done once, at a single point in time to help to make that decision. Other organizations might decide that long-duration type programs, ongoing efforts, need to be constantly or periodically monitored to ensure that they are still delivering the desired for outcomes or that processes associated with them are not deteriorating in their effectiveness.

Surveys can be used for both single points in time and on-going long duration reviews as well as to measure program processes, content, and outcomes and there are appropriate uses to all in program evaluation.  Some of the questions that program evaluation efforts can be used to answer include:

Process

  • Is the program being carried out well, and how can it be improved?
  • How can you measure the impact of various programs on organizational processes?

Content

  • Are the people targeted by the program understanding, retaining and actually utilizing the content being provided?
  • Is the content covering all critical areas?

Outcomes

  • Will the program have an impact on organization effectiveness, increasing the organizations ability to perform?
  • Which programs are the most cost effective, providing the biggest bang for the buck?
  • How can I measure the return on investment (ROI) of various programs?

© 2010 by Jeffrey M. Saltzman. All rights reserved.

Visit OV: www.orgvitality.com

Written by Jeffrey M. Saltzman

November 21, 2009 at 7:50 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: