On April 16, 2020, SCEA Board Members and Members convened at a virtual town hall to share experiences, best practices, and lessons learned about evaluation practice in the context of COVID-19. With 25 participants spanning educational, healthcare, public and social services, and international sectors, we discussed key challenges and needs for evaluators. We’ve compiled a summary of key takeaways from this call.

The impact of COVID-19 on the field of evaluation. COVID-19 is impacting evaluation practice in many ways, especially for international evaluations requiring travel, nonprofit and informal programs where evaluators are experiencing job insecurity, the logistics of doing our evaluation work, and programs simply no longer being implemented due to social distancing. Moreover, research and evaluation might not be viewed as a priority right now. 

It is our duty to maintain ethical evaluation practices during this time. We must consider the human experience and the ethics of data collection and evaluation practice. Acknowledging the present state helps to provide empathy and sensitivity. Beginning a survey email invitation with a statement such as “We understand that the current times are difficult and uncertain” can go a long way. Similarly, consider beginning an interview with a check-in rather than delving straight into the interview protocol.

Check in with clients and hear their concerns and needs. Clients have many program-level concerns. Take a step back, bring levity and logic to the evaluation setting, and help explain what your evaluation is providing at this time. What data collection items are mandatory? What future concerns exist that need solution brainstorming now? What data must be collected now to complete mandated reporting, and what data collection items can be put on hold until programs or events are implemented as originally intended? 

Understand that data collected may be very “time-and-place” specific. Participant experiences are not similar to anything prior and it will be difficult to compare program experiences in spring 2020 to spring 2019. Consider what data collection activities are necessary and can be scaled back if they do not provide a lens on a “typical” implementation. However, also consider the opposite — how can we use data from this to examine deeper institutional or contextual issues and needs? 

Look into alternate forms of data collection. Consider phone surveys and shortening online surveys, or shifting from surveys to qualitative data collection to gather more personalized depth to the experience. In some cases, you may need to provide longer interviews to allow interviewees more time to check in and express concern over current conditions. 

Across the board, the COVID-19 pandemic is affecting all evaluators and impacting the role of evaluators in each sector a little differently. For those in education, it is our role as evaluators to be able to tell the story. For those in public or social services, we evaluators must ask ourselves: What are we responding to? How well did all of it work? And how do we know it worked?

Continue connecting with evaluation peers and colleagues. We were happy to be able to share a time and space for Southern California evaluators to discuss key challenges in the context of COVID-19. Continue connecting with other evaluators by joining SCEA, listening in on AEA Coffee Break events, following #EvalTwitter on Twitter, and joining discussion forums and “un-webinars” at EvalCentral. 

Lastly, a few resources shared by participants during our town hall: 

• Best Practices Conducting Phone Surveys
• ASSESS Listserv
• Webinar: Discussion on Challenges and Strategies for M&E in the Time of COVID-19 by the US Agency of International Development (USAID) and Implementer-Led Design, Evidence, Analysis and Learning (IDEAL)
• Upcoming Events from the Food Security and Nutrition Network (FSNN)