Impact- if not measured or evaluated correctly, can be very difficult to understand and quantify. In this blog, we will see what an impact evaluation study is, what are the steps involved in completing an evaluation study, and how to communicate the study results to the stakeholders involved.
An ideal program or intervention functions with a goal and certain objectives in mind. Impact evaluations are necessary to gauge to what extent have the intended impacts of the intervention been achieved. They provide information about not just the intended objectives but also the unintended impacts. Impact evaluations give an understanding of how well a program has worked, and what are the areas that can be strengthened in it.
Let’s look at the steps for an effective impact evaluation study.
Step 1: Framing research questions
Research questions are questions that shape data collection, analysis, and reporting.
| Process evaluation | Impact evaluation |
| Frame questions about the implementation of the project. Inputs, outputs, program activities, and stakeholder views will be included in this. | Frame questions about the changes created by the project. These can be short-term changes like knowledge, skills, and attitudes; medium-term changes like behavior and actions; and long-term changes like conditions and status. |
Evaluate your research question. Is it feasible? Is it interesting? Is it novel? Is it ethical? Is it relevant?
Step 2: Indicator development
An indicator is a variable that measures the key elements of a program. An indicator has 4 parts.
| Population | Who is changing? |
| Target | How many do we expect will succeed? |
| Timeline | By when does this need to happen? |
| Threshold | How much is good enough? |
Make sure that the indicators you choose are easy to understand and will provide essential metrics to determine impact. They need to be cost-effective and fit within the program budget. Use a combination of quantitative and qualitative indicators. Only include indicators that are aligned with the program components and goals. Experts can be involved to get their insights on it.
Step 3: Sampling
Sampling is the process of selecting a set of observations from a population to estimate a program’s impact, for that population. The chosen sample should be a fair representation of the population. The types of sampling are:
| Probability Sampling | Non-Probability Sampling | ||
| Simple random | Every single member of a population is chosen randomly | Purposive | Sample is selected purely considering the purpose of study |
| Systematic | Members of a sample are chosen at regular intervals of a population | Convenience | Used in studies which need to reach subjects which are difficult to trace |
| Stratified | The population is divided into smaller groups, that don’t overlap | Snowball | Dependent on the ease of access to subjects |
| Cluster | The entire population is divided into sections or clusters | Quota | Selection of members happens on basis of a pre-set standard |
| Proportion to percentage | Random sampling techniques to select a sample from subpopulations | Voluntary response | Made up of participants who have voluntarily chosen to participate |
Step 4: Research tools
The research design and tools need to be decided prior to the start of data collection. These tools should focus on garnering data that will provide rich insights into the external factors which enable or deter the achievement of impact and program objectives. Make sure each question adds value and drives survey responses that relate directly to the research goals and evaluation questions. Keep wording focused on the respondent and their opinions, rather than introducing a point of view of your own. Undertake a pilot study to understand possible response biases.
Step 5: Field research
A work plan needs to be made which sets out the various stages of data collection, analysis, and deadlines for the various deliverables. A qualified team needs to be chosen and trained to conduct the field research. There are several tools like Open Data Kit, SurveyCTO, KOBO, Google forms, Magpie, Track you Metrics, etc to collect data. Supervisors need to ensure the quality of data on the field along with the enumerators and monitor the data regularly.
Step 6: Reporting
The report of an impact study should cover the objectives of the evaluation and should be easy to understand. It should reflect the purpose and answer the key questions of the research. The components of a report are:
- Cover Page
- Table of Contents
- List of Figures and Tables
- List of Abbreviations
- Executive Summary
- Introduction
- Methodology
- Results and Findings
- Summary of Results according to OECD DAC
- Recommendations
- List of Annex
Step 7: Sharing findings with stakeholders and using the results for strategy
Just providing a comprehensive report to the stakeholders is not enough, communicating it in the best way possible is important too. The findings of the reports can be shared through presentations, summarised versions of the report, recorded videos, etc. Use existing online tools and software to create engaging and interesting content. Guide the stakeholders to use the findings of the report to bring about the necessary changes to the program.
- Give real-time results to stakeholders to help them start working on it from the beginning.
- Recommendations must be prioritized within a time frame to the extent possible so that the next measures can easily be taken.
- The learnings must be incorporated into the overall programming cycle through an effective feedback system.
