The need for both evaluators and commissioners to engage effectively with stakeholders: issues, strategies and challenges
Chih Hoong Sin, Head of Information and Research, Disability Rights Commission;
Phil Copestake, Deputy Head of Analytical Studies Unit, Office for Public Management;
Richard Watts, Delivery and Planning Manager, Disability Rights Commission,
Reflect on joint learning about what worked in the evaluation process.
Introduction
The Disability Rights Commission (DRC) of Britain was established by an Act of Parliament and came into existence in 2000. It ceased to exist with effect from 1 October 2007, with the setting up of the new Equality and Human Rights Commission (EHRC). Given the short time span of the DRC’s existence, its Senior Management Team (SMT) and commissioners decided that there was a need to evaluate the extent to which the DRC has had an impact in the areas identified in its Strategy, and whether there were any lessons that were transferable to the EHRC and other disability organisations.
An evaluation of the DRC’s impact was commissioned in 2005, with the aim of evaluating:
- the impact of the DRC’s work to promote and implement rights, enforce rights, and influence law and policy;
- the impact of the DRC’s activities on closing the gaps of opportunity between disabled and non-disabled people; and
- the processes and structures underpinning the implementation of these activities and the extent to which they mediate or mitigate against the effectiveness of the DRC’s activities.
The Office for Public Management (OPM), an independent, employee-owned research and consultancy organisation, was commissioned to conduct the evaluation. While the evaluation report, published in September 2007, contained useful information relating to how things worked in terms of the DRC’s activities, it became clear towards the end of the evaluation that there was a separate but related issue of what worked in the process and practice of the evaluation itself that merited investigation.
Stakeholder engagement
The complexity of the evaluation and the multitude of different requirements of the evaluation outputs meant there were a number of individuals with a stake in how the evaluation was conducted, what outputs were produced, when outputs were required and in what format they were presented. Effective stakeholder engagement was thus paramount in ensuring that the processes and outputs of the evaluation were fit for purpose.
Two specific dimensions of stakeholder engagement were identified by OPM and DRC in post-project review as being of particular significance. The first related to the evaluators’ (OPM’s) engagement with the commissioner of the evaluation (DRC). This is often described in terms of ‘client relationship management’, the basic tenets of which are relatively well documented; however, specific issues may arise in relation to particular evaluations and so the relative importance of individual factors will vary.
The second dimension of effective stakeholder engagement identified by the team is often overlooked. This relates to the need for commissioners of evaluations to engage effectively with a range of internal stakeholders. There can be an implicit assumption that the client is a monolith to be ‘engaged with’ by evaluators. This posits commissioners of evaluations as (a) internally undifferentiated entities with a single, well-defined need; and (b) passive and reactive to external impetus for ‘engagement’.
Effective stakeholder engagement needs to be in place at the commencement of an evaluation, developed throughout its duration, and extended after its completion. It is common to find that both evaluators and clients generate their own (often implicit) understandings of what worked well within the evaluation without having these understandings verified by the other party. Assumptions can thus be unfounded or not specific enough to draw out actionable learning. Embracing the evaluative spirit, the DRC and OPM decided to work together to identify what worked in terms of stakeholder engagement within this evaluation. Our intention was to highlight how post-evaluation engagement can identify useful lessons that contribute towards improving practice.
The journey
In embarking on a journey of learning after the evaluation, we decided it would be informative for us (the ‘evaluators’ and the ‘commissioners’) to generate – independently and without prior discussion – our own ideas of ‘what worked’, ‘why it worked’, and what we felt worked less well and why. The ‘view from DRC’ is shown in Table 1 and the ‘view from OPM’ in Table 2 below.
Once these views were captured, we circulated our respective sets of notes to the other party in early October 2007. We then had sufficient time to digest the information before a meeting was set up in late October 2007 to discuss the issues and to distil key themes and lessons.
Co-creating an understanding of effective engagement
It is clear from the perspectives described in Tables 1 and 2 that, while there were broad overlaps in the ways in which DRC and OPM viewed effective engagement, there were also differences between the two perspectives.
A meeting was held in late October 2007 involving the three authors of this article to distil some key themes from the two different perspectives.