Archive for the 'Program Management Topics' Category

Some Myths About Program Evaluation

1.. Many people believe evaluation is a useless activity that generates lots of boring data with useless conclusions. This was a problem with evaluations in the past when program evaluation methods were chosen largely on the basis of achieving complete scientific accuracy, reliability and validity. This approach often generated extensive data from which very carefully chosen conclusions were drawn. Generalizations and recommendations were avoided. As a result, evaluation reports tended to reiterate the obvious and left program administrators disappointed and skeptical about the value of evaluation in general. More recently (especially as a result of Michael Patton’s development of utilization-focused evaluation), evaluation has focused on utility, relevance and practicality at least as much as scientific validity.

2. Many people believe that evaluation is about proving the success or failure of a program. This myth assumes that success is implementing the perfect program and never having to hear from employees, customers or clients again — the program will now run itself perfectly. This doesn’t happen in real life. Success is remaining open to continuing feedback and adjusting the program accordingly. Evaluation gives you this continuing feedback.

3. Many believe that evaluation is a highly unique and complex process that occurs at a certain time in a certain way, and almost always includes the use of outside experts. Many people believe they must completely understand terms such as validity and reliability. They don’t have to. They do have to consider what information they need in order to make current decisions about program issues or needs. And they have to be willing to commit to understanding what is really going on. Note that many people regularly undertake some nature of program evaluation — they just don’t do it in a formal fashion so they don’t get the most out of their efforts or they make conclusions that are inaccurate (some evaluators would disagree that this is program evaluation if not done methodically). Consequently, they miss precious opportunities to make more of difference for their customer and clients, or to get a bigger bang for their buck.

Michael Quinn Patton


Program Evaluation

Programs should be evaluated on at least a yearly basis to discern if the programs are reaching their goals, achieving their outcomes and if they are doing so in an efficient manner. Small businesses seldom have the resources to conduct evaluations of a program’s goals, outcomes and process. However, they can think about where they have the most concerns about a program and then gear an evaluation to look at that aspect of the program.

Program evaluation holds numerous advantages. It can verify or increase the results/outcomes on customers (internal or external). It can fine tune delivery of program services, which, in turn, saves costs and time. Evaluations often provide wonderful testimonials that can be used for public relations and credibility of the program. In fact, evaluations are often used by program planners to ensure that the program is indeed carrying out the original process planned for the program in the first place. Often, the program plan ends up changing dramatically over time as program employees are overcome by events. Program processes can naturally deviate from the original plan because program plans were flawed in the first place, the program’s environment changed a great deal or program employees simply found a much better way to deliver products/services to customers (internal or external). For more information about program evaluations, their planning and how to carry them out.

Goals-Based Evaluation

Often programs are established to meet one or more specific goals. These goals are often described in the original program plans.

Goal-based evaluations are evaluating the extent to which programs are meeting predetermined goals or objectives. Questions to ask yourself when designing an evaluation to see if you reached your goals, are:
1. How were the program goals (and objectives, is applicable) established? Was the process effective?
2. What is the status of the program’s progress toward achieving the goals?
3. Will the goals be achieved according to the timelines specified in the program implementation or operations plan? If not, then why?
4. Do personnel have adequate resources (money, equipment, facilities, training, etc.) to achieve the goals?
5. How should priorities be changed to put more focus on achieving the goals? (Depending on the context, this question might be viewed as a program management decision, more than an evaluation question.)
6. How should timelines be changed (be careful about making these changes – know why efforts are behind schedule before timelines are changed)?
7. How should goals be changed (be careful about making these changes – know why efforts are not achieving the goals before changing the goals)? Should any goals be added or removed? Why?
8. How should goals be established in the future?