Students who earn their graduate certificate in Project Monitoring & Evaluation from American University emerge from the program equipped with the skills to conduct their own evaluations. They learn how to design evaluations, how to collect and analyze data, and how to write quality evaluation reports. They learn how to be evaluators.

The work that I do as a measurement, evaluation, and learning officer at the Gordon and Betty Moore Foundation puts an interesting spin on the traditional evaluator role, since I am not conducting evaluations myself. At the Moore Foundation, we routinely commission external consultants to conduct evaluations of the work that we fund. In my role, I identify and select evaluators to work with, collaborate with them to design evaluations, and manage them throughout the evaluation process.

Part of this work involves drafting requests for proposals that are distributed to prospective evaluators, reviewing the proposals that are submitted, and ultimately selecting one. There are several criteria that I use when reviewing the proposals from prospective evaluators. Below are some of the criteria that I believe are the most important.

• How well the evaluator understands the field and the program’s theory of change. I work specifically with the Moore Foundation’s science division, which supports discovery-driven, non-biomedical research. Our science programs span a breadth of disciplines, ranging from marine microbiology to quantum physics. Therefore, it is critical that an evaluator demonstrate a strong understanding of the field, and I assess the degree to which the proposal is tailored to the content of the program. In addition to an understanding of the field, a proposal must demonstrate that the evaluator has a good grasp on the program’s theory of change.
• Team composition and expertise. In their proposals, prospective evaluators should include a description of the composition of the evaluation team, respective roles, and bios for the different team members. I look for a proposed team that is well-rounded in terms of the roles they will have in the upcoming evaluation, as well as their areas of expertise. We encourage the use of mixed methods in the evaluations we commission, so it is important that experience in both qualitative and quantitative methods are represented among the proposed team. In addition, it is critical that at least some of the evaluation team members have backgrounds that are relevant to the scientific discipline of the program being evaluated.
• Access to relevant stakeholders. While it’s important to have evaluation team members with science backgrounds, I also recognize that there are few (if any) evaluators out there who also happen to be marine microbiologists, for example. In a strong proposal, an evaluator will demonstrate that they know how to engage domain experts in the evaluation, as needed. For example, an evaluator might suggest enlisting a small group of marine scientists as advisors to the evaluation process. Does it seem like the evaluator has a good understanding of the right type of people to reach out to? Do I have confidence that the evaluator will be compelling when they ask these stakeholders to participate? These are both questions that I ask myself as I review a proposal.
• Project management plan. Poor project management can threaten the quality and rigor of an evaluation. For this reason, I ask prospective evaluators to include a description of their project management plan in their proposals. Has the evaluator thought through how they will keep the evaluation on schedule and on budget? When will they submit deliverables, and what is the appropriate sequence of deliverables? What is the evaluator’s protocol for communicating with the foundation? How will they report on progress? How will the evaluator handle any changes or obstacles that arise while they are conducting the evaluation? A strong project management plan should address each of these questions.
• Quality of past evaluations. Strong proposals are further strengthened by a track record of high-quality evaluations. For this reason, I request that prospective evaluators submit samples of their past work. Evaluators should try to submit examples that they feel are as similar as possible to the work they anticipate doing for the upcoming evaluation (both in terms of similar methodology and relevant content and evaluate). This helps to demonstrate not only that the evaluator is capable of conducting high-quality evaluations, but also that they are able to sufficiently familiarize themselves with relevant scientific disciplines.

Some evaluators have a tendency to use the same set of proposed methods for data collection and analyses regardless of the request for proposal (RFP) that they’re responding to. “Cookie cutter” proposals are unlikely to be successful. Rather, a strong proposal demonstrates that the evaluator has thoughtfully determined which methods they believe will be best suited for the objectives of that particular evaluation. Being creative with the evaluation design, while also adhering to monitoring and evaluation best practices, will certainly make a proposal stand out.

Selecting a strong evaluator is only the first step in successfully managing an evaluation, and, even though I am not conducting evaluations myself, I have learned that the same rigor that one would use when carrying out an evaluation must be brought to the process of managing one.

About the Author

Julia Klebanov is a measurement, evaluation, and learning officer at the Gordon and Betty Moore Foundation. Julia’s work is specifically focused on the measurement, evaluation, and learning practices of the Moore Foundation’s Science Program. Prior to her current role, Julia held the position of program manager for the Science Program. Julia holds a B.A. in Biology and a minor in Chemistry from Oberlin College, a certificate in project management from Stanford University, and a graduate certificate in Project Monitoring and Evaluation from American University.

To learn more about American University’s online MS in Measurement and Evaluation or Graduate Certificate in Project Monitoring & Evaluation, request more information or contact an admissions advisor at 855-725-7614.