In practice, however, the projects are rarely subjected to systematic and technically defensible impact evaluations. Feb 25, 2009 common sense evaluation has a very long history, evaluation research which relies on scientific methods is a young discipline that has grown massively in recent years spiel, 2001. The evaluation experience is likely to be more positive andits results are likely to be more useful if you build evaluation in from the start andmake it an ongoing activity. Impact evaluation allows one to conclude authoritatively, whether or not the. Jun 10, 20 difference between inputs, activities, outputs, outcomes and impact some of the most confused terminologies within the research, monitoring and evaluation field are the words, input, activities, outputs, outcomes and impact.
This means that an impact evaluation must establish what has been the cause of observed changes in this case impacts referred to as causal attribution also referred to as causal inference. Model of undp contribution to outcomes and impact 14 figure 3. Review your logic model to think about each of your proposed outcomes or objectives, and impact. Impact evaluation evaluation of the effect outcome changes of behaviour and energy saved. According to the world bank oassess changes in the wellbeing of individuals, households, communities or firms that can be attributed to a particular project, program or policy oprovides feedback to help improve the design of programs and policies question. Impact evaluation and study designs to measure effectiveness. Comparing process and outcome evaluations national council. Appendix 3 implications of complication and complexity for impact evaluation. An impact evaluation provides information about the impacts produced by an intervention positive and negative, intended and unintended, direct and indirect. It does not provide definitive evidence, however, that the observed outcomes are due to the intervention. Counterfactual analysis enables evaluators to attribute cause and effect between interventions and outcomes. In an assessment, the relationship between assessor and assessee is reflective, i. This includes planning the summative evaluation beforethe intervention begins as part of the planning process, which helps to clarifyprogram goals and reasonable outcomes.
After providing a definition for outcome indicators, as well as a list of motivations for the new emphasis on outcome section 2, two basic proposals are made for the commission to formally adopt in its regulation. Stakeholdersincluding program directors, community members, and funderswant to know if a particular program is working as expected and making a difference for its intended participants. Impact or outcome evaluations are undertaken when one wants to assess. An impact evaluation relies on rigorous methods to determine the changes in outcomes which can be attributed to a specific intervention based on causeandeffect analysis. Impact evaluation evaluation of the effectoutcome changes of behaviour and energy saved. Although evaluation designs may produce useful information about a programs effectiveness, some may produce more useful information than others. Abstract impact evaluation provides information about the impacts produced by an intervention. Jun 14, 2018 outcome evaluations assess the effectiveness of a program in producing change. Choosing appropriate designs and methods for impact evaluation. Outcome evaluations assess the effectiveness of a program in producing change. The results chain see below attempts to categorise these steps by breaking them down into manageable stages inputs, activities, outputs, outcomes and impact. Very few programs have the same effect for everyone and it is rare that changes introduced to a school or a system will simply work or not work in a binary fashion. Impact evaluations need to account for the counterfactual what would have occurred without the intervention through the use of an experimental or quasiexperimental design using comparison and treatment groups. A proposed model to conduct process and outcome evaluations and implementation research of child health programs in africa using integrated community case management as an example theresa diaz, 1 tanya guenther, 2 nicholas p oliphant, 1 maria muniz, 1 and the iccm symposium impact outcome evaluation thematic group.
Additionally, observations and analysis by the evaluation team were added to the challenges and adaptations that were documented by this project, based on its own internal monitoring and evaluation. Summative evaluation an overview sciencedirect topics. Outcome evaluations or impact evaluations focus on the questions that ask what happened to program participants and how much of a difference the program made for them. Evaluation is a systematic process to understand what a program does and how well the program does it. This method was utilized in the example of the towns drug prevention program. The main types of evaluation are process, impact, outcome and summative evaluation. Implications for sustainability were developed and included, along with recommendations for program activities and components. Process evaluations help stakeholders see how a program outcome or impact was achieved.
Instead, it is important to give adequate consideration to the various aspects of. What constitutes credible and appropriate impact evaluation. Impact evaluations seek to assess broad, longterm changes that occur as a result of a program. An impact evaluation measures the programs effects and the extent to which its goals were attained. Types of evaluation centers for disease control and. Difference between inputs, activities, outputs, outcomes and. An impact evaluation approach which unpacks an initiatives theory of change, provides a framework to collect data on immediate, basic changes that lead to longer, more transformative change, and allows for the plausible assessment of the initiatives contribution to results via boundary partners. Formative evaluation process evaluation outcome evaluation. Session 1 impact evaluation methodologies impact evaluation workshop, lilongwe, dec.
This involves trying to measure if the program has achieved its intended outcomes. Purpose of impact evaluation impact evaluation serves both objectives of evaluation. Please note that different classifications exist but the taxonomy is not important in. Impact evaluation assesses the changes that can be attributed to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones. Local health departments all over california participate in the impact outcome evaluation project. Impact evaluation assesses program effectiveness in achieving its ultimate goals. By using impact evaluations, we know how effective our programs are and can make good, costeffective decisions about future programs to reduce extreme poverty and boost shared prosperity. An impact evaluation provides information about the impacts produced by an intervention positive. A process evaluation describes a programs services, activities, policies, and procedures. An impact evaluation determines the causal effects of the program.
An impact evaluation may, for example, show that lower rates of communitywide solid waste accumulation were the direct result of a school ee program promoting community recycling and composting behaviors. Differences in focus, scope and purpose between project and outcome evaluations 3. Prepost design 10 other methodological issues in outcome evaluation 12 1. Although not a steadfast rule, most jurisdictions should be able to make some assessments about program impact after 12 months of activities. The latter state that while the outcome evaluation tells us what kind of change has occurred, an impact evaluation paints a picture as to how a program might have affected participants lives on a broader scale. Impact or outcome evaluations are undertaken when it is important to know whether and how well the objectives of a project or program were met. As a complex system, delivering impact is not so different. Assessment is concerned with process, while evaluation focuses on product. The first one concerns the methodological principles to be met by outcome indicators section 3. Impact evaluations sometimes alternatively called outcome evaluations, gather and analyze data to show the ultimate, often broader range, and longer lasting, effects of a program. Overview of impact evaluation page 2 an impact evaluation should only be undertaken when its intended use can be clearly identified and when it is likely to be able to produce useful findings, taking into account the availability of resources and the timing of decisions about the programme or policy under investigation. How to evaluate the programme pdf, 262kb world health. Outcome evaluation seeks to provide a nuanced view of the program, exploring who the program had an impact on, to what extent, in what ways and under what circumstances. Outcome evaluation is a method of determining a programs effectiveness by measuring results.
For example, an impact evaluation might assess the impact of a development project or programme. Outline of principles of impact evalation discussion dra. Impact evaluations need to account for the counterfactual what would have occurred without the intervention through the use of an experimental or quasiexperimental design using. Two common ways to assess a program are process evaluation and outcome evaluation. The handbook provides an overall context for resultsbased monitoring and evaluation and. Most often this form of evaluation is used to measure whether or not a technology such as hie. For example, designs that track effects over extended time periods time series. Guidelines for outcome evaluators 5 introduction this publication, guidelines for evaluators, is the first in the companion series to the handbook on monitoring and evaluating for results. Outcomes are the shortterm and intermediate changes that occur in learners, program participants, visitors, etc. Methodologically, randomized evaluations are the best approach to estimate the effect of a program 1. In this video, learn about the types of program evaluation, how to focus your evaluation design, and different options to conduct impact and outcome evaluations.
Effectiveness and efficiency relating evaluation criteria to results categories 16. The ltpr impact evaluation tool is designed to evaluate the outcomes and impacts of land. The increasing emphasis on impact evaluation in development has led to many questions. This project helps identify successful nutrition education interventions and potential best practices, while providing direction for program improvement. Outputs, outcomes and impact are terms that are used to describe changes at different levels from the delivery of goods and services to longterm, sustainable change in peoples lives. Understanding different types of program evaluation brad.
Impact evaluation can be undertaken of a programme or a policy, or upstream work such as. Outcome effectiveness evaluation measures program effects in the target population by assessing the progress in the outcomes or outcome objectives that the program is to achieve. Evaluation terminology goal what you hope to achieve or accomplish objective specific and measurable steps to accomplishing goals output measurable result of a program activity outcome what changed or was gained as a result of a program activity indicator used to monitor progress in achieving outcomes and impact. It provides early feedback as to whether the program is being implemented as intended, what barriers have been encountered, and what. Sep 30, 2016 in this video, learn about the types of program evaluation, how to focus your evaluation design, and different options to conduct impact and outcome evaluations. The purpose is to achieve a higher level result say, a. Introduction 2 since the focus of impact evaluation is on outcomes rather than process, we reorganize the projects by the following key outcome categories. It can be undertaken of a programme or a policy, or upstream work such as capacity building, policy advocacy and support for an enabling environment. Outcome and impact measures outcome measures assess the extent to which the objective is immediately met always involve some level of change in attitude, knowledge, behavior have a baseline or benchmark impact measures designed to assess impact of the program or service over time almost always include a comparison group. The table below shows the results chain using a set of definitions originally developed by oecd dac in 2002 oecd, 2010. In contrast to evaluation, in which the feedback relies on the level of quality as per set standard.
Some of the most confused terminologies within the research, monitoring and evaluation field are the words, input, activities, outputs, outcomes and impact. Ohi aims to impact different levels of an organisation, often referred to as combined, individualorganisational or multilevel interventions. Outcomeeffectiveness evaluation measures program effects in the target population by assessing the progress in the outcomes or outcome objectives that the program is to achieve. Page 2 of 6 evaluation terminology goal what you hope to achieve or accomplish objective specific and measurable steps to accomplishing goals output measurable result of a program activity outcome what changed or was gained as a result of a program activity indicator used to monitor progress in achieving outcomes and impact benchmark standards by which. Understanding these terminologies not only ensures that appropriate indicators are identified, but also that they are. In contrast to outcome monitoring, which examines whether targets have been achieved, impact evaluation is structured to answer the question. Variations in timing, scope, purpose and duration of outcome evaluations 2. The key questions that the evaluation will want to address is. An outcome evaluation tells you whether a program achieved its goals. Evaluation and implementation research of the delivery of maternal, newborn and child health interventions that focus on impact, specifically those assessing changes in morbidity or mortality, have been deemed critical to determine the effectiveness of programs being implemented. Outcome evaluation focuses on the effect the training had on learners and their organizations. Compared to the combined outcome of activities contributing to a purpose, impacts. In other words, process evaluations document the process of a programs implementation. Choosing an impact evaluation method 187 determining which method to use for a given program 187 how a programs rules of operation can help choose an impact evaluation method 188 a comparison of impact evaluation methods 193 finding the smallest feasible unit of intervention 197 chapter 12.
The focus of a process evaluation is on the types and quantities of services delivered, the beneficiaries of those services, the resources used to deliver the services. However, unlike an efficacy trial, in reality evaluators and researchers cannot control. Evaluating early childhood education and care programs. Evaluation is an important part of managing a successful program. Summative evaluation is intended to identify possible problems, assess performance of members, seek resolution to problems found, and appropriately adjust the strategic planning process to meet current needs and future expectations.
The purpose is to achieve a higher level result say, a reduction in child mortality. A proposed model to conduct process and outcome evaluations. Outcome evaluation outcome evaluation assesses the effectiveness of a program in producing change. Credible and appropriate impact evaluation can greatly improve the effectiveness of development. Impact evaluation and study designs to measure effectiveness marie ng assistant professor, global health. Such research indicates the degree to which the intended outcomes occur among the target population. Below, some examples of evaluation objectives for process and impact evaluation are given. Comparing process and outcome evaluations national.
In this regard, the cpo evaluation model distinguishes the levels of individualsleaders and groupsorganisation cf. The impact evaluation should be based on data accumulated for at least 12 months of program activities. What is the difference between an impact and an outcome. The demonstration sites were required to submit their evaluation reports in their 18th month.