‘Invisible hands’ in the evaluation process – Commissioning, Resourcing and Procuring evaluations

There is a growing realisation among sections of the global evaluation community that we do not direct our critical gaze enough at key aspects of the functioning of the evaluation system itself nor, what might be termed the sociology of evaluative practice.  By this we mean the array of interests, social dynamics and social/political structures which characterise evaluation as a social practice. Key aspects of this include a glaring gap around processes of commissioning evaluations. The way evaluations are commissioned, how they are procured and how they are paid for, paradoxically given their importance, have not had a large amount of attention (notwithstanding some exceptions[1]) The effects of the unconscious systemic bias of these practices have long been experienced by those undertaking evaluations in the global south

RTE 2 focuses  on the commissioner and the attendant processes of commissioning evaluations, resourcing, and procurement .

Whilst experiences differ, there are some discernible tendencies .

• Many evaluation commissioners are primarily, and in some cases solely, incentivized by accountability pressures, rather than learning for change or transformation
• Commissioners are limited and defined by those who have the funds to undertake an evaluation,
• International aid has skewed its part of the market, as it has resources for evaluations for its own accountability and learning purposes, and has dominated the field in commissioning evaluations of activity in Global South
• Methods are presented as monolithic, rather than nuanced
• Commissioners routinely underestimate the power, technical capacity and authenticity of local or indigenous knowledge, rendering it invisible (some commentators have called this epistemic justice),
• Terms of reference skills demands are ‘expert or technically oriented’ leading to a lack inclusive evaluation designs.
• Evaluations are undertaken by large consultancies
• Perceived restrictive procurement systems reduce freedom to choose an evaluation team which can deliver best
• Evaluation system is virtually a closed circuit ie from Commissioner and back to the same Commissioner for decision-making.

The RTE will bring together  panellists who have thought about these issues and more. They will exchange experiences and their informed views and offer concrete suggestions to make evaluative practice fit for the future.

[1] See for example Jones, Lindsey, Laura Kuhl, and Nathanial Matthews. “Addressing power and scale in resilience programming: A call to engage across funding, delivery and evaluation.” The Geographical Journal 186.4 (2020): 415-423.

Cox, Jayne, and Pete Barbrook-Johnson. “How does the commissioning process hinder the uptake of complexity-appropriate evaluation?.” Evaluation 27.1 (2021): 32-56.R

Round Table Panellists

Hala Elsayed

Hala Elsayed is a career civil servant, and per passion for working on complex policy evaluations has grown throughout her career.  She currently leads the Experimentation and Evaluation Hub in the Ministry of Justice, that focuses on strategically increasing the capacity, capability and quality in the use of experimentation and evaluation across the Department.  She is an economist by profession and currently a member of the Trial Advice Panel and a Council Member of the UK Evaluation Society.  She has led on economic appraisal and policy evaluations for a range of policy areas including vocational education, business support and more recently court reform.  She is particularly driven by using a systems approach to explore complex policy issue.

Dr Rachel Iredale

Independent Evaluation Consultant, UKES Member

Rachel Iredale works as a Consulting Director in the Dublin office of RSM Ireland providing specialist evaluation expertise to both internal and external stakeholders.  She has an extensive background in research and consultancy services, working in a range of sectors and across a variety of disciplines.  A champion of using evaluation for continuous improvement and to inform decision-making, Rachel has designed, commissioned and delivered evaluations in areas as diverse as health and social care, education, construction, and technology, and is always interested in exploring innovation in method.

Rachel was previously Head of Evaluation, with overall responsibility for the evaluation portfolio at the Economic and Social Research Council (ESRC) and Evaluation and Impact Manager at the Construction Industry Training Board (CITB).  Other roles have included being a Director of Services for a Welsh cancer charity and directing a 10 year programme of research with patients and the public on genetics.

Rachel has a PhD in social policy from University College Dublin and a Masters in European Social Policy Analysis, having spent time studying at Maynooth University of Bath and the University of Tilburg.  Currently her main research area is exploring innovative methods, such as Citizens’ Juries, that involve the public in policy making as a Visiting Fellow for Public Engagement at the University of South Wales.

David Rider Smith

David Rider Smith is Senior Evaluation Coordinator, Evaluation Service of UNHCR.  David leads on building the decentralized evaluation function in UNHCR and is the deputy to the Head of Evaluation.  He has over twenty years of experience in evaluation, research and performance monitoring, including roles as head of evaluation capacity and quality at the UK Department for International Development (DFID); as an advisor to the Government Evaluation Facility, and at the Uganda Bureau of Statistics.  David has worked in the UN system previously, for two years as the senior officer and team leader in the Joint Inspection Unit of the UN system, and leading a UN system-wide study on change management (2018-19) and at the UNDP Evaluation Office as an Evaluation Specialist (2003-07).  He has a strong record in research and publications worked for three years at the Water, Lands and Ecosystems Research Programme of the CGIAR (2014-17) and four years at the Natural Resources Institute (1999-2003).

Moderator : Professor Murray Saunders

Murray is Professor of Evaluation in Education and Work at Lancaster University.  He runs a PhD module on evaluative practice, is assistant editor of Evaluation and has undertaken a wide range of evaluations and written on inclusive approaches to evaluation, capturing effects of interventions and the consideration of the usefulness of evaluations.  He adopts a social practice approach to evaluation.  Murray is a founding member of the UK Evaluation Society and an ex-President also of the European Evaluation Society and part of the formation group of the IOCE and Evalpartners.