Expanding the Reach of Impact Evaluation (ERIE)

Date Range: 2016 - 2025Project Lead: Pulte InstituteNotre Dame Collaborators: Center for Research ComputingContact: Danice Guzmán

Piloting a new evaluation methodology: Retrospective Long-term Impact Evaluations

International development programs are designed to improve health, education, and financial well-being for disadvantaged populations. However, it can be extremely difficult to measure the long-term impact of such programs. Long-term program evaluations are unlikely due to rigid funding cycles which prioritize evidence generation at the end of the life of a project and not beyond. This means that evaluations typically occur just a few years after an intervention has launched. As such, evaluations tend to focus on the immediate and rarely measure the results of a policy or program over long periods of time.

This knowledge gap poses a formidable dilemma for development organizations, depriving them of needed evidence.This is particularly true in sectors where theories of change extend far beyond the conventional project cycle. The absence of proof for sustained impact hinders the ability to make strategic and cost-effective choices.

In 2016, Notre Dame’s Pulte Institute for Global Development partnered with AidData, the Center for Effective Global Action, Mathematica Policy Research, and a group of geospatial researchers led by Mark Buntaine at UCSB, to launch a new consortium: Expanding the Reach of Impact Evaluation (ERIE). 

ERIE was an approach to conducting retrospective long-term impact evaluations of completed USAID interventions. ERIE evaluations leverage and build on existing data from two sources: programmatic data outlining where, when and how programs were implemented, and geocoded secondary data including satellite imagery or living standards surveys, to rigorously assess impact in the long-term. The ERIE consortium also created a guide on planning for and conducting these long-term impact evaluations, which could be incorporated into USAID bureau and mission strategy.

Prior to the cancellation of this award, ERIE completed nine pilot studies:

  1. Malawi: ERIE conducted a retrospective geospatial impact evaluation of Food for Peace’s Wellness and Agriculture for Life Advancement (WALA) activity in Southern Malawi. Using qualitative and quantitative methods, the team examined the long-term impact of WALA on agriculture, nutrition, resilience, and sustainability outcomes. Despite the positive results found at endline, the evaluation found that effects were not sustained in the long term. (Research team: Pulte, Mathematica, AidData)

  2. Israel/West Bank: ERIE retrospectively evaluated the Office of Conflict Management and Mitigation’s People to People (P2P) activities. Primarily using a qualitative method with a short online survey, the evaluation explored the potential effects of previous P2P projects on perceptions of and continued communication with those on the other side of the conflict and belief in the potential of peace or co-existence. The study found lasting and positive effects of perceptions of and communication with the other group; however, findings were more mixed regarding the potential for peace and reconciliation. (Research team: Pulte)

  3. Uganda: ERIE conducted a retrospective evaluation of Feed the Future’s Commodity Production and Marketing Activity, implemented from 2013-2018. This program aimed to improve the value chain by facilitating interactions between exporters, traders, and village agents. This ex-post evaluation found activities were sustained, and spillover continued among other value chain actors. (Research team: Pulte)

  4. Peru: ERIE conducted a follow-up to a randomized control trial evaluation of Amazonia Lee, an intervention in the Amazon region of Peru that focused on capacity building of teachers to improve early grade reading performance. The follow-up aimed to understand if short-term impacts for second-grade students were sustained into the fourth grade. The study found that the impacts were not sustained and offered recommendations on how to better support students in later grades. (Full report available here) (Research team: Mathematica)

  5. Honduras: ERIE conducted a seven-year qualitative follow-up to a randomized control trial evaluation of the Edducacción-PRI intervention, which focused on providing more data to schools and teachers through the use of formative assessments and end-of-grade assessments. The follow-up study demonstrated widespread support for the continued use of these tools, but also shed light on some of the challenges to accessing them, such as limited resources and delayed data delivery to teachers. (Research team: Mathematica)
  6. Globally: A long-term impact evaluation of the Center for Development Research's Partnership for Enhanced Engagement in Research (PEER) examined the impact of USAID’s funding for scientists on research capacity, evidence generation and policy change. Researchers traveled to Uzbekistan, Colombia, and Morocco to interview PEER/HESN researchers in 2019-2020 and conducted interviews of scientists via Zoom in South Africa and Mozambique during the pandemic. Other data collection methods included an online survey, and a web-scraping exercise to collect bibliometric information of all PEER scientist applicants. We found that research grants positively impact researcher’s career progress, and have a short term impact on the reach of the scientists’ publications. We find limited results in terms of translation from research to policy change. (Full report available here). (Research team: Pulte, ND Center for Research Computing)

  7. Globally: The mixed-methods long-term performance evaluation of USAID’s Higher Education Solutions Network (HESN) activities analyzed survey, bibliometric, administrative, interview, and focus group data. Results showed that HESN-supported research and innovations Labs played a vital role in advancing universities’ efforts to achieve HESN’s outcomes and goals in collaboration with USAID, policymakers, and other partners. (Research team: Mathematica)

  8. Bangladesh: ERIE conducted three endline evaluations of the Bureau for Humanitarian Assistance’s Food for Peace activities in three regions of Bangladesh: Nobo Jatra, implemented in Khulna and Sakhira Districts of Southwest Bangladesh; SHOUHARDO III, implemented in the char and hoar wetland regions of Northern Bangladesh; and SAPLING, implemented in the Chittagong Hill Tracts of Bangladesh. ERIE evaluators conducted pre-post analysis, paired with qualitative data collection and quasi-experimental methods. The programs were found to impact some outcomes, including dietary outcomes, antenatal care, and some resilience indicators. However, ERIE could not detect impacts in terms of childhood stunting and other health and nutrition outcomes compared with similar communities. (Research team: Pulte, Mathematica, AidData)

At the time of cancellation, several other ERIE evaluations were underway across many sectors and geographies:

  1. Domestically: ERIE was assessing the long-term contribution of the Payne Fellowship Program towards the goal of a more diverse and inclusive Foreign Service Workforce at USAID. ERIE was also analyzing the cost effectiveness of the Payne Fellowship as a pathway for recruiting and hiring new Foreign Service Officers from traditionally underrepresented groups, and assessing the performance of Howard University as an implementer of this program over 13 years. (Research team: Pulte)

  2. Bosnia-Herzegovina, Colombia, Nigeria, and Zimbabwe: ERIE was conducting a long-term impact evaluation of the Center for Violence Prevention’s People to People programming, on outcomes such as attitudes towards and friendships with outgroups. Using a matching methodology, a conjoint experiment and a qualitative comparative method, this study examined the long-term impact of ten peacebuilding interventions across many different conflict contexts. (Research team: Pulte)

  3. Ethiopia and South Sudan: ERIE was conducting ex-post evaluations of emergency crop activities using several methods: In Ethiopia, a geospatial quasi-experimental evaluation aims to understand the long-term effects of a potato intervention implemented by the International Center for the Potato. In South Sudan, UND Professor Cat Bolten piloted the use of the ethnographic method to understand the long-term effects of an emergency seed intervention implemented by CARE-International. An evidence review preceded both of these studies, assessing the state of the evidence on the long-term effects of emergency crop interventions as a humanitarian tool. These evaluations were funded by USAID’s Bureau of Humanitarian Assistance (BHA). (Research team: Pulte, Mathematica, AidData)

  4. Peru: ERIE was conducting two geospatial retrospective impact evaluations on environmental and sustainable development activities, funded directly by USAID Peru. ERIE was evaluating the Recuperación de Ecosistemas en Comunidades Kechwa y Awajun (RECONKA) activity which aimed to help Indigenous communities plan and coordinate their internal land use, as well as the Alternative Development (AD) activities, which aimed to support farmers in their transition from illicit crops to other profitable crop cultivation. (Research team: Pulte, UCSB, ECU)

  5. Globally: ERIE was conducting a long-term evaluation of USAID’s Digital Strategy efforts. Included in this long-term evaluation was a survey of digital stakeholders from more than 30 countries, a deep dive on several countries which implemented the Digital Ecosystem Country Assessments, and a geospatial impact evaluation of a WomenConnect Challenge recipient, the GeoKrishi app, in its effort to close the gender digital divide in Nepal. (Research team: Pulte, Mathematica, AidData)

  6. Globally: ERIE was conducting a long-term evaluation of USAID’s Exploratory Programs and Innovation Competitions (EPIC). This evaluation would have assessed the long-term impact of receiving a prize for innovators on follow-on funding, jobs, revenue or profit, as compared with other applicants who did not receive the cash prize. (Research team: Pulte, CEGA)

 


‹ All Pulte Institute Projects