The Power to Commission: Exploring the Evaluation Capacity and the Transformative Role of Evaluation Commissioners in the International Development Sector

<p>Evaluation commissioners play a central role in evaluation practice by generating demand for evaluation through issuing calls for evaluation services and managing evaluation contracts (Lemire, Nielsen et al., 2018; Nielsen et al., 2018b), particularly in the international development sector. Yet few competency sets specify the knowledge and skills needed to conduct this work. Even less is known about the evaluation marketplace in the international development sector and the contract mechanisms used to procure evaluation services.</p> <p>This research explored the capacity to commission evaluation among international nongovernmental organization (INGO) evaluation commissioners, including individual competencies and practices for commissioning an evaluation, factors that facilitate and constrain the commissioning process, and ways in which evaluation commissioners can contribute toward decolonizing evaluation. The research questions were answered using an explanatory sequential mixed methods design consisting of two phases: (a) a quantitative phase (survey) followed by (b) a qualitative (multiple case study) phase comparing and contrasting INGOs with emerging and high evaluation capacity.</p> <p>Phase One of the research included a survey of INGO evaluation commissioners (<em>N</em> = 93) to understand individual and organizational evaluation capacity, commissioning practices and influential factors, and awareness of calls to decolonize evaluation. Respondents possessed a median of 10 years of commissioning experience and reported (on a five-point scale) high levels of individual evaluation capacity (<em>Mdn</em> = 4.5, IQR = 4.3, 4.6). Organizational evaluation capacity varied by organizational resources for evaluation and organizational use of evaluation. On average, respondents rated the INGO capacity to use evaluation (<em>M</em> = 3.7, <em>SD</em> = 0.64) slightly higher than organizational resources provided for evaluation (<em>M</em> = 3.4, <em>SD</em> = 0.75).</p> <p>Commissioners reported always or often recruiting external evaluators by publicly posting requests for proposals (67.6%, <em>n</em> = 54), by circulating among a short-list of evaluators (33.8%, <em>n</em> = 27), or by referrals (30.1%, <em>n</em> = 24). Three-quarters of respondents (<em>n</em> = 62) reported seeking evaluation expertise as the top qualification when recruiting evaluators; yet two-thirds (<em>n</em> = 53) reported they did not use an evaluator competency framework when reviewing evaluator qualifications. Over 60% (<em>n</em> = 48) of respondents reported being moderately to very aware of calls to decolonize evaluation. However, 16.6% (<em>n</em> = 12) reported contributing a lot or a great deal to these efforts.</p> <p>Phase Two consisted of a multiple case study of four INGOs with differing levels (emerging, high) of organizational evaluation capacity by evaluation resources and evaluation use. Interviews (<em>N</em> = 21) and document review (<em>N</em> = 36) were analyzed by case. A cross-case synthesis illuminated that the evaluation commissioning process is relatively standard across INGOs but that interpersonal, organizational, and systems-level factors influence commissioner decision making and the commissioning process.</p> <p>Interpersonal factors included coordination and communication not only with evaluators but with internal and external interest holders (<em>N</em> = 4 cases). Organizational factors included organizational evaluation policies (<em>N</em> = 4 cases), organizational culture (<em>n</em> = 3 cases) and structure (<em>n</em> = 2 cases), and INGO staff evaluation capacity (<em>n</em> = 2 cases). Systems-level factors included the evaluation marketplace (<em>N</em> = 4 cases), donor requirements (<em>n</em> = 3 cases), and the nature of the development sector (<em>n</em> = 3 cases).</p> <p>Commissioners had varying levels of awareness and agreement with calls to decolonize evaluation. Some articulated ways to contribute to decolonization by contracting with local evaluators, adopting participatory approaches to engage local communities, and using evaluation methodologies indigenous to the local context.</p> <p>The collective findings illuminate the state of the field regarding evaluation commissioning, identify strengths and challenges in the commissioning process, and point to future directions for strengthening this capacity and contributing to decolonizing evaluation within the international development sector.</p>

Name thesis:degree_name

Psychology, PhD

Level thesis:degree_level

Open Access Dissertation

Discipline thesis:degree_discipline

School of Social Science, Politics, and Evaluation

Year dc:date.available

2025

Author dc:creator

• Castleman, Ann Marie

Contributors dc:contributor

• J. Bradley Cousins
• Michelle Sloper
• Steffen Bohni Nielsen

• evaluation capacity
• evaluation commissioner
• international development
• international nongovernmental organization
• research on evaluation
• International Relations
• Psychology

Repository record dc:identifier

https://scholarship.claremont.edu/cgu_etd/1031

OAI identifier oai:identifier

oai:scholarship.claremont.edu:cgu_etd-2052