Trends in Monitoring and Evaluation: Self-Reflection, Privilege, and Culturally Responsive Evaluation

I am a 2019 graduate of American University’s Measurement and Evaluation Program. I entered the program with US Peace Corps experience and was working at USAID/Zambia on HIV/AIDS programming at the time. As I continued my coursework with the program, I learned different evaluation theories, approaches, and designs, and became more confident in my abilities to evaluate HIV programs. I considered myself a person who was conscious of my status and privilege in the evaluation process, and my coursework reinforced the need to be self-reflective.

When my employer won a contract to evaluate the impact and cost effectiveness of Complementary and Integrated Health services (i.e. massage, acupuncture and chiropractic services), I approached the evaluation protocols similarly to my HIV evaluation work. However, I did not consider how my privilege as a person without disabilities would greatly impact the way that I approached this evaluation. I wanted to share the lessons that I have learned as my evaluation practice evolves.  

Lesson 1: Understanding privilege

We need to be self-reflective in our evaluation practice; and consider all of our privileges. I recognize that I have an advantage because I am a person without disabilities. Sometimes people without disabilities perceive themselves as “normal,” and wrongly presume that everyone has the same opportunities, abilities, and access.  As an evaluator, I learned that it is important to understand my privileges and reflect on these as they relate to my work. This means that I need to craft evaluation designs and instruments that are sensitive to people with disabilities (more on that below).

Lesson 2: Collaborate with stakeholders

Talking to stakeholders with disabilities can help us become familiar with the community and assess the community’s readiness and willingness to address the issues that the evaluation may raise. The community leaders can share history, attitudes, culture, and values, that may affect the level of stakeholder buy-in. As evaluators, we need to familiarize ourselves with the cultural context and setting of each program we evaluate. In the case of my project, my team and I met multiple times with the advisory board to receive feedback on our interview protocols and surveys. Their insight was invaluable as they were able to help us gauge the efficacy of the evaluation instruments and receive feedback on the appropriateness of the questions. It was also helpful to ensure that our evaluation was Utilization-Focused and centered on the needs and decisions of the intended users.

Lesson 3: Ensure that surveys are accessible for all members of the population

Survey accessibility is something that is relatively new for me. I have always understood it in the sense of monitoring the jargon used in surveys and translating surveys into different languages. However, due to my previous lack of awareness I never thought about accessibility for people with disabilities. According to Alchmer (Formerly Survey Gizmo), there are two requirements for a survey to be accessible:

  1. Content that may be difficult for people with visual impairments to navigate or that screen readers cannot interpret should be excluded.
  2. Necessary messaging should be added to ensure that all people can understand how to use the survey.

My team and I have decided to administer the survey in three modalities: email, phone, and mail. With this, we hope that the survey is being administered in ways that will assist all people responding to it.

Self-reflection and my evaluation practice

I am evolving everyday as I continue to work with people with disabilities. However, I find it necessary to share my experiences with you as you might be assigned an evaluation with a population with which you are unfamiliar. It is important to acknowledge and check your privilege at every step of the evaluation. Empathy and open mindedness have helped the evolution of my evaluation practice, as I respond better to the needs of people with disabilities.

About the Author

Tiffany Smith, M.S., is a Research Analyst at Mission Analytics Group, Inc. with extensive project management, HCV experience, and HIV experience both domestically and internationally. At Mission, Ms. Smith serves as Project Manager of the Hepatitis C Medicaid Affinity Group (Affinity Group), a collaborative working group that brings together state public health and Medicaid programs to improve treatment access for HCV. In this role, she organized and arranged travel for an in-person convening, coordinates and facilitates monthly webinar meetings, and helps to analyze outcome measures to assist states in developing accurate estimates of HCV burden. Ms. Smith has also contributed to Various HRSA HAB projects. She serves as an analyst on the Data and Reporting Technical Assistance (DART)/Data Integration, Systems, and Quality (DISQ) team where she has provided proactive and warm-line TA to RWHAP recipients and providers to ensure high data completeness and quality.). She served as a Project Manager for the HRSA HAB Formula Calculations Methods Assessment Project where she spearheaded the environmental scan of various federal formula grant programs, synthesized the results from the scan into a concise database, and ensured the on time, high quality submission of all project deliverables. Prior to joining Mission, as a contractor for the United States Agency for International Development (USAID), Ms. Smith served as the technical lead for all site visits. As the site visit lead, she created all data collection tools and conducted key informant interviews, focus groups, and interviews with HIV health care providers and government officials to gather the information that would assist in the increase in the quality of HIV services. Lastly, Ms. Smith has extensive research and evaluation experience. As a member of the evaluation staff of the Emory Rollins School of Public Health COMPASS (Committed to Partnerships in Addressing HIV/AIDS in Southern States) Initiative, she created interview protocols, conducted research of organizational best practices in addressing HIV/AIDS and cost-effective programs, wrote internal reports, and developed internal and external standard operating procedures. Ms. Smith holds an M.S. in Measurement and Evaluation from American University.