Promoting Community Resilience: Impact Evaluation of the Disaster Response Program
In 2022, Global Ministries and the United Methodist Committee on Relief, UMCOR contracted the Pulte Institute’s Evidence and Learning Division to perform an impact evaluation for its United States Disaster Response, USDR program. The overall purpose was to assess and capture the outcomes of the program’s strategies and interventions and identify best practices.
From 2017 to 2020, the USDR program conducted extensive disaster relief and recovery efforts through five United Methodist Church Annual Conferences (Florida, Mississippi, South Carolina, North Carolina, and Alabama-West Florida) and the Methodist Church of Puerto Rico, benefitting over 8,000 households and mounting to $56 million in funding.
The primary objective was to examine the recovery experiences of those communities and assess the USDR program’s role in supporting them during their recovery process, inform future programming, and contribute to the larger body of knowledge in hurricane recovery services. Disaster resilience is a broad concept, making measurement and evaluation efforts challenging. Most studies focus on measuring the factors underpinning resilience, such as institutional capacity and government effectiveness; however, Pulte Institute’s Evidence and Learning Division believes developing better strategies requires a more systematic evaluation.
Using a mixed-methods approach, the Pulte Institute’s Evidence and Learning Division assessed the USDR program’s disaster relief and recovery efforts and overall success using the OECD’s six evaluation criteria: impact, relevance, coherence, effectiveness, sustainability, and efficiency.
The evaluation—overseen by the Pulte Institute’s Paul Perrin, Sisi Meng, and Lila Khatiwada—consisted of a mixed-methods approach:
· A quantitative data-collection method involving household and comparison household surveys.
· A qualitative data-collection method consisting of focused group discussions and key informant interviews.
Based on the evaluation results, the research team recommended improving the program’s effectiveness to serve as a roadmap for future programming and decision-making.
The findings are assisting Global Ministries and UMCOR in rethinking their case management approach to prioritize activities that have the most significant long-term impact on the recovery of affected communities. They are also helping Global Ministries and UMCOR to develop new models and incorporate new technologies to aid in disaster planning and resource allocation, including platforms that can be used to gather real-time information from affected individuals to optimize the distribution of food, water, medical supplies, and personnel.
Broader Implications
By understanding Global Ministries and UMCOR’s impact and genuinely engaging with those served, it is necessary to ask, “What are we doing, and are we achieving what we think we are? If not, what can be done differently?
The Pulte Institute’s research study is helping to reframe strategies to focus on things that can have a meaningful impact — instead of just hoping efforts are successful. The findings are valuable in shaping a better response to future hurricanes. For example, one recommendation is to build evaluation into relief efforts so that it is part of the process. Going forward, Global Ministries and UMCOR will actively collect data and information during hurricane relief efforts to inform long-term planning and resilience-building measures.
The effectiveness of new models and technologies depends on factors like data availability, integration with existing systems, and coordination among faith-based organizations, government agencies, and the private sector. However, data is only helpful if it is used and useable and helps make good decisions.
The research study’s findings also call attention to the importance of looking at social and demographic data to help identify vulnerable populations, such as older adults or those with disabilities, who may need additional support during and after a hurricane—and communicating accurate and timely information to the public through various channels, including social media.