Expanding the Reach of Impact Evaluation (ERIE)

A Toolkit for Retrospective Long-term Impact Evaluations

International development programs are designed to increase respect for human dignity through improvements in health, education, and financial well-being for disadvantaged populations. However, it can be extremely difficult to measure the long-term impact of such programs. Long-term program evaluations are often constrained by rigid cycles, precarious funding streams, and ever-shifting priorities. As such, they tend to focus on the immediate and rarely measure the results of a policy or program over long periods of time and large geographic scales.

This knowledge gap poses a formidable dilemma for development organizations and actors, depriving them of the essential insights needed to navigate the intricate decision-making landscape. This is particularly true in sectors where theories of change extend far beyond the conventional project cycle. The absence of proof for sustained impact becomes a stumbling block, hindering the ability to make strategic and cost-effective choices and perpetuating a void in our understanding of the enduring footprint of development interventions.

In 2016, Notre Dame’s Pulte Institute for Global Development partnered with AidData, the Center for Effective Global Action, Mathematica Policy Research, and a group of geospatial researchers led by Mark Buntaine at UCSB, to create a toolkit: Expanding the Reach of Impact Evaluation (ERIE).

ERIE is an approach to conducting retrospective long-term impact evaluations of completed USAID interventions. These evaluations will leverage and build on existing program data to assess if the observed short-term impacts are sustained or to investigate results that might only be expected to emerge over a long-term horizon. ERIE uses innovative data collection strategies and methods to identify the appropriate counterfactuals and generate lessons on planning for and conducting these long-term impact evaluations, which can be incorporated into USAID bureau and mission planning. 

ERIE has completed nine pilots:

1. In Malawi (final report): ERIE conducted a retrospective longitudinal impact evaluation of Food for Peace’s Wellness and Agriculture for Life Advancement (WALA) activity in Southern Malawi. Using qualitative and quantitative methods, the team examined the long-term impact of WALA on agriculture, nutrition, resilience, and sustainability outcomes. Despite the positive results found at endline, the evaluation found that effects were not sustained in the long term.

2. In Israel/West Bank (final report): ERIE retrospectively evaluated the Office of Conflict Management and Mitigation’s People to People (P2P) activities. Primarily using a qualitative method with a short online survey, the evaluation explored the potential effects of previous P2P projects on perceptions of and continued communication with those on the other side of the conflict and belief in the potential of peace or co-existence. The study found lasting and positive effects of perceptions of and communication with the other group; however, findings were more mixed regarding the potential for peace and reconciliation.

3. In Uganda (final report): ERIE conducted a retrospective evaluation of Feed the Future’s Commodity Production and Marketing Activity, implemented from 2013-2018. This program aimed to improve the value chain by facilitating interactions between exporters, traders, and village agents. This ex-post evaluation found activities were sustained, and spillover continued among other value chain actors.

4. In Peru (policy brief): ERIE conducted a follow-up to a randomized control trial evaluation of Amazonia Lee, an intervention in the Amazon region of Peru that focused on capacity building of teachers to improve early grade reading performance. The follow-up aimed to understand if short-term impacts for second-grade students were sustained into the fourth grade. The study found that the impacts were not sustained and offered recommendations on how to better support students in later grades. (Full report available here)

5. Globally (policy brief): A long-term impact evaluation of the Center for Development Research's Partnership for Enhanced Engagement in Research (PEER) examined the impact of USAID’s funding for scientists on research capacity, evidence generation and policy change. Researchers traveled to Uzbekistan, Colombia, and Morocco to interview PEER/HESN researchers in 2019-2020 and conducted interviews of scientists via Zoom in South Africa and Mozambique during the pandemic. Other data collection methods included an online survey, and a web-scraping exercise to collect bibliometric information of all PEER scientist applicants. We found that research grants positively impact researcher’s career progress, and have a short term impact on the reach of the scientists’ publications. We find limited results in terms of translation from research to policy change. (Full report available here).

6. Globally (full report): The mixed-methods long-term performance evaluation of USAID’s Higher Education Solutions Network (HESN) activities analyzed survey, bibliometric, administrative, interview, and focus group data. Results showed that HESN-supported research and innovations Labs played a vital role in advancing universities’ efforts to achieve HESN’s outcomes and goals in collaboration with USAID, policymakers, and other partners.

7. In Bangladesh: ERIE conducted three endline evaluations of the Bureau for Humanitarian Assistance’s Food for Peace activities in three regions of Bangladesh: Nobo Jatra, implemented in Khulna and Sakhira Districts of Southwest Bangladesh; SHOUHARDO III, implemented in the char and hoar wetland regions of Northern Bangladesh; and SAPLING, implemented in the Chittagong Hill Tracts of Bangladesh. ERIE evaluators conducted pre-post analysis, paired with qualitative data collection and quasi-experimental methods. The programs were found to impact some outcomes, including dietary outcomes, antenatal care, and some resilience indicators. However, ERIE could not detect impacts in terms of childhood stunting and other health and nutrition outcomes compared with similar communities.

Seven other evaluations are underway globally across many sectors: diversity, equity and inclusion, conflict and violence prevention, early-grade reading, emergency crop interventions, environmental and forestry activities, digital strategy, innovation, and prizes. Using ERIE, retrospective evaluations can occur either when short-term impacts have been assessed but not tracked over time or when interventions are only likely to have impacts over longer periods of time. ERIE also created a guide to help evaluators and USAID offices assess whether a long-term evaluation is feasible for a given program and to integrate plans for long-term evaluations into future programming. This guide helps create low-cost opportunities for long-term impact evaluation by providing recommendations on collecting and compiling certain types of program data in systematic ways from the initial stages of measurement.

‹ All Pulte Institute Projects