This blog will take you through the implementation basics of an SIA. You can read more about the technical aspects of an SIA, in Part 1 of this blog series.
What is the process of doing an Social Impact Assessment (SIA)?
- Define the research questions and objectives for your SIA: At the outset, identifying what question/s your research will attempt to answer is imperative. For example, do you want to determine if your planning or strategy is appropriate for your target group, or if your implementation is as per estimation or plan? Do you intend to manage your intended and unintended impacts through the SIA? This process of defining your research questions is directly aligned with key facets of your research- Sampling, Tools, Analysis and Reporting
- Identify your stakeholders and key respondents: Determining the units of analysis and the right persons/stakeholders to collect data from is the next step. For example, if you are looking at teacher training programmes, you may want to look at implementation from the teacher’s end and impact at the student level.
- Detail what kind of data you need to collect: Based on research questions and type of respondents, aligned indicators must be identified and/or developed. These indicators should be contextually-informed and SMART (Specific, Measurable, Attainable, Relevant, Time-based). In addition, targets and benchmark values must be determined to assess success and gaps during the analysis process.
- Identify the right tools: Identifying whether you want to do telephone interviews or Focus Group Discussions or in-person observations or surveys or ethnographic research will depend on your area of research, sampling, and indicators i.e. which data source is most appropriate.
- Undertake the data collection: An understanding of available data and data gaps must first occur. In addition, logistic and operational aspects must be determined, and potential risks must be anticipated, with mitigation strategies in place. Once the data collection process is underway, checks and balances must ensure that the data is reliable, accurate, and valid.
- Analyse the data: Once the data has been collected, the analysis process must be matched to the indicators using quantitative software such as Excel, SPSS, etc. Qualitative findings must be used to interpret the quantitative insights.
- Use the results: Along with systematic documentation of the analysis and consultations undertaken, there has to be a strategy to use and disseminate the results. This should be informed by the audience of the results, the relevance of different results for different audiences, and the channels of communication.
What are the challenges one encounters when doing an SIA?
- Research objectives: Evaluators must avoid the risk of trying to do too much and asking too many questions when the funds and/or time may be constrained. Evaluators should focus on utilising the OECD DAC criteria of relevance, efficiency, effectiveness, sustainability, and coherence.
- Indicator development: Too many indicators and fuzzy indicators could affect the quality of the collected data. In addition, greater reliance on perception skills or focusing on outputs rather than outcomes and results could limit the value of data. This dearth of knowledge around the understanding of indicators and how to measure them could be addressed by getting consultants or experts to look at this process in a more scientific manner.
- Sampling: One main challenge relates to attrition rates. For instance, a programme may have 500 people at baseline and be left with 200 participants at the endline. Or, if a programme is doing Zoom meetings or using pictorial tools, there could be a bias in the sample. You should choose a larger sample set in your baseline while simultaneously identifying mitigation strategies by using technology and tracking platforms.
- Data collection and analysis: Whilst doing an SIA, if concerns like resources, budgets, capacity, and proficiency are not accounted for, it could affect the kind and quality of data that the SIA is able to collect. An absence of baseline data or over-reliance on quantitative data may constrain the full picture of impact.
- Data analysis and reporting– Writing a long, cumbersome report and falling into the trap of reporting everything that has been found through the SIA, will limit the views and utility of the SIA findings. Rather, identifying 3-4 relevant points for different stakeholders, along with a good balance between text and visualisations will help the findings reach the intended audiences.
What are some examples of SIAs that 4th Wheel has carried out?
4th Wheel believes in a multi-dimensional approach. While we also look at cross-cutting issues like women empowerment, food security, gender equality, we mainly carry out assessments in the education, health, and livelihoods spaces.
Education: We have evaluated several education-related projects, disaggregated at the school system, school administration, teachers, and students’ levels. We have been involved with program evaluations spanning teacher training, infrastructure development, digital access, sports for development and other relevant spaces. For instance, we carried out a longitudinal study for children from 9th to 11th standards who participated in a career counselling programme. Our assessment indicated that participation in career awareness programmes led to improvements in their aspirations, reduced drop-outs, and defined career goals.
Health: We have looked at behaviour change in the health space. With regard to maternal and neonatal health, we have looked at interventions at the hospital, community, and individual level. For instance, in one study, we found that a large proportion of new mothers who received an ante-natal information booklet did not use it. In comparison, a video, which was also more cost-effective, was more likely to be viewed.
Livelihoods– We have done more than 15 evaluations to assess the impact of skilling on income and employment levels, and how this relates to change in status and family life. For instance, an impact evaluation of a skill development programme indicated that participating women experienced improvements in confidence and self esteem, and were able to define clearer life goals.
If you are looking to carry out a Social Impact Assessment, reach out to us at hello@the4thwheel.com.
4th Wheel is running the Explained series. In this series, we take up emerging approaches/ tools/ methodologies and break them down for ease in understanding.
Works Cited
Social Impact Assessment Guidelines: For State Significant Projects. New South Wales Department of Planning, Industry and Environment. 2021. https://www.planning.nsw.gov.au/Policy-and-Legislation/Under-review-and-new-Policy-and-Legislation/Social-Impact-Assessment
Kohn Parrott, Kate, and Roger Panella. “A Learning Guide to Social Impact Assessment for Community Health: SIX STEPS TO MEASURE OUTCOMES AND DEMONSTRATE SUCCESS.” Greater Detroit Area Health Council. https://www.choosingwisely.org/wp-content/uploads/2019/08/GDAHC_Learning-Guide.pdf
“Social Impact Assessment.” International Union for Conservation of Nature. 2016. https://www.iucn.org/sites/dev/files/iucn_esms_sia_guidance_note.pdf
“Social Impact Assessment.” Inter-American Development Bank. 2018. https://publications.iadb.org/publications/english/document/Social_Impact_Assessment_Integrating_Social_Issues_in_Development_Projects.pdf
“Impact Assessment”. International Federation of Red Cross. https://www.ifrc.org/docs/evaluations/impact-handbook.pdf
“MEASURING UP: Landscaping the State of Impact Assessment Practices amongst Corporate and Family Foundations in India”. Das
This blog is written by Nandita Palrecha, Associate- Monitoring and Evaluation at 4th Wheel Social Impact.
