Blog from Gordon Freer Commissionership – Community of Practice 1

Evaluation commissioning plays a crucial role in optimising the purpose and use of evaluations.  However, it is often overlooked as a specialist area. To address this and its impact on the larger evaluation ecosystem, UK Evaluation Society has created a six-month pilot, community of practice for commissioners and evaluators to discuss their experience and perceptions of issues in the commissioning process, with a view in the first instance to exchanging ways to improve them. See the earlier blog on ‘Commissionership’

The first community of practice session focused on the common frustrations of commissioners and evaluators, which arise at various points in the evaluation cycle, from developing terms of reference to sharing results. The next ‘Commissionership’ community of practice session is scheduled February 16th 1300 -1400 GMT on Purpose and Audience for an evaluation. Register here.

Some of the issues which arose in this first discussion include:

Commissioners need to balance the interests and priorities of stakeholders in an evaluation – especially with internal colleagues.
Commissioners and evaluators shared difficulties in their evaluation experiences, including the need to balance the interests and priorities of stakeholders, especially those of internal colleagues. These frustrations occur at various points in the evaluation cycle, such as the development of Terms of Reference, identifying the end user of the evaluation, the evaluation process itself, the approval of results, and the distribution of findings.
Each of these points in the evaluation cycle had its own context and stresses. In initiating an evaluation and finalising a Terms of Reference, sometimes evaluations are commissioned because of an unplanned or unexpected event. While it is easy to suggest that all projects are implemented with an evaluation in mind, and due consideration and time is allocated to defining the purpose, function, and scope of an evaluation, this is not always possible. As a result, calling for an evaluation in short order might result in different interests coming to the fore. These interests might not always be complementary, pertinent, or valuable to the project or the evaluation. Competing interests and trying to be “all things to all commissioners” results in a far broader scope, forgoing a more in-depth, focused evaluation.

Scheduling of an Evaluation – timing critical to subsquent use
Timing evaluations also frustrates commissioners. It is recognised that there may be a need to allocate resources to a specific evaluation within a period. However, the resource cycle and the project implementation cycle are not always aligned, sometimes causing the scheduling of an evaluation that is either too early to too late. The results is an evaluation sharing findings that have limited use for the project and the commissioner.

Resource Commitment of the intervention itself – how much is sought?
Related to the timing of an evaluation is the extent and level of resource commitment by the intervention or project itself in the evaluation. A key consideration discussed was how involved do we, the commissioners, want to be in the intervention or project, and will this take away from the main purpose of delivery?

Commissioners request for ‘innovative approaches’ – what is behind this?
A popular call amongst commissioners to request innovative approaches from evaluators was identified as a frustration, reflecting interests from internal stakeholders, or even different understandings of the term. ‘Innovation’ might be requested because commissioners are not sure which methodology would be most appropriate, or commissioners are not always aware of new methods being developed and refined. Alternatively, commissioners might really be looking for
innovation in the evaluation space. Of course, innovation requires an appetite for risk, and risk carries with it a possibility of failure. While it is easy to call for innovation, not many commissioners want to balance this request with a failed evaluation.

Sharing Results of an Evaluation – Which results? With whom? What format?
Both commissioners and evaluators, raised the frustration of sharing the results of an evaluation.  Commissioners raised the questions with whom do we share the results, and in what format are these shared? Do we share results that show failure, or less than perfect performance? Many expressed a reluctance of their organisations to share failures, or often to share results externally, in spite of the commissioners themselves recognising the value of learning from these. Related to
sharing these results, do commissioners request long, detailed reports that may go unread, or shorter, punchy slide decks and infographics, that might focus on highlights, but appear less than robust? And yet sharing pertinent points, in accessible format, with identified constituencies can provide a solid foundation for organisational learning.

Two considerations
This first Community of Practice session suggested two areas to consider addressing some of these frustrations.
1. To be more open about failure and shortcomings, recognising that failing, and admitting failure, are essential steps to improvement and growth
2. To initiate early and frequent conversations between commissioners and evaluators as partners – or collaborators – rather than adversaries.

Members, come along and join the next ‘Commissionership’ community of practice session, scheduled February 16th 1300 -1400 GMT on Purpose and Audience for an evaluation. Register here.

If you are a commissioner, evaluator, or in procurement of evaluation, and would like to join the planning group for the series, we want to hear from you – do get in touch with hello@evaluation.org.uk Attn: Commissionership