Blog from Isabella Di Paolo Commissionership – Community of Practice 5

The fifth Community of Practice in the UK Evaluation Society’s pilot series on ‘Commissionership – towards better practice’ focused on the use of evaluation findings and recommendations.

The session involved a mix of commissioners and evaluators, and several participants with experience in both roles. The session focused on discussing evaluators’ and commissioners’ roles and responsibilities in ensuring findings, and recommendations are relevant, implementable, and potentially impactful.

The usefulness of an evaluation is determined by its ability to inform decision-making. Despite this, creating conditions for the use of evaluation findings comes with important challenges, including contrasting project timelines and priorities. The timelines for decision-making and for evidence generation do not always exactly align, and relevant decision-makers are not always sufficiently engaged throughout the evaluation process for it to be as productive as intended.

The discussion focused on potential ways in which evaluators and commissioners can work together, to help ensure that evaluations lead to evidence-based programming in practice. These included:

Sharing emerging findings at different points during the evaluation cycle. This was seen as a potential way to support evidence-based programming and increase client engagement and understanding. However, this needs to be done carefully, so as not to lead to decisions
being made on incomplete data. It is also crucial to ensure all potential users are aware of the ‘draft’ status of the findings, and what this means for decision-making.   Examples were shared, including where things had gone awry – for instance, where some stakeholders had
treated draft findings as final and already shared them externally, before the final version had been written and months before the commissioner had accepted the final evaluation.

Having open lines of communication between commissioners and evaluators during the evaluation process was seen as key from both parties. It is helpful to establish regular moments of contact and be available for any ad hoc issues which may arise.  On the one
hand, it is important to ensure commissioners and evaluators have similar expectations on what the evidence will be able to say (and inform). It is also important to strike a balance between maintaining evaluator’s independence, and fostering collaboration. Ultimately, it is
important for the evaluator to have a clear understanding of the audience of the evaluation, as well as their expectations, plans and vision for using the findings, so that the evaluator is well placed to ‘land’ the evaluation findings and recommendations.  In other words, so that
the evaluator can write the evaluation findings and recommendations in such a way that they will be received/’heard’ by those they are directed towards.

Planning ahead is necessary to ensure that the contract includes sufficient time and resources for uptake and dissemination activities. Dissemination should not be an afterthought, but clearly planned in from the outset by both evaluators and commissioners. Evaluators work and products can also assist commissioners to support learning after the evaluation timeframe. Commissioners have an important role in guiding evaluators to understand the audience and what will be useful for them and securing the buy-in of key decision-makers, including identifying ‘allies’ who will be able to influence or take up findings.

Useful pointers to consider when commissioning or leading evaluations include:

Transparency, clarity, and shared understanding are key – for both parties. For evaluation findings to be useful, and used, it is crucial to understand the evaluation’s end users, including the level of complexity which would be suitable for them.

• Commissioners can help evaluators by supporting them to understand organisational needs and dynamics.
• Having a transparent and structured process to both respond to and implement findings and recommendations (such as management response systems) can help support learning and evidence-driven iteration.  (Recognise also the danger that this becomes a ritual, it needs to be meaningfully undertaken).
• Commissioners should identify support/individual allies for the findings and recommendations among their organisational leadership and decision-makers, as they are key to influencing any wider changes recommended.

Evaluations should move from pure contracts, to being collaborations. As highlighted in several previous sessions, the partnership nature of conducting evaluations was also emphasised in this discussion. Although evaluators and commissioners have different roles and responsibilities, it is important to build on each others’ strengths, working together to improve accessibility and use of evaluations, and ultimately to foster evidence-driven decision-making. In practice, this means that:

• Evaluators should focus on delivering high-quality and clear products written/presented in a way which stakeholders will receive (or ’hear’) the findings and recommendations.
• Evaluators can be part of leading dissemination of findings and recommendations and working with the commissioner to ensure the dissemination can continue once the evaluator’s contract is over.
• Commissioners have a role to play in ensuring the appropriate people are engaged throughout the lifetime of the evaluation, so that they gain some depth and insight into the issues and evidence which produced the findings and recommendations, and are potentially in a better position to support them.
• Ultimately, a collaborative style contract is built on a high degree of trust and open communication.

Building engagement and ownership of the client, is a shared responsibility for evaluators and commissioners. Although it is not possible to ensure that findings will necessarily be used, there are several strategies which can create the conditions for this to take place. Commissioners and Evaluators can design an evaluation process so that it builds engagement and ownership of an evaluation – an important starting point for eventual use.  A couple of strategies highlighted include

• Lead co-creation and sensemaking workshops. These can make it possible to work with the client to identify feasible and relevant recommendations.
• Share emerging findings to help build engagement, but it comes with some risks. The correct timing and having shared expectations are both important, to avoid either sharing already known information, or insufficiently validated recommendations.

The session identified some actions to take to improve practice:

• Commissioners need to reckon on the process of using evaluation findings and recommendations to make change as a connected part of an evaluation exercise. Ie The evaluation report is not the end, it is only halfway through. Commissioners need to make time and structure processes for dissemination of information to reach appropriate postholders/constituencies after the evaluation products are available.
• Commissioners have an important role to play in opening space within their organisations to foster learning and evidence-based decision-making, involving the appropriate people at the right times and increasing their interest and ownership of the evaluation, both during and after the evaluation exercise, with a view to enhancing take-up of findings and recommendations.
• Commissioners should set out the balance of engagement in the relationship between themselves and evaluators in the Terms of Reference, and in contractual arrangements, this includes the level of engagement they want from the evaluator in dissemination of findings and recommendations, and the initial stages of influencing any changes. Evaluators should feel free to challenge or negotiate some variation as they see fit.
• Evaluators should feel empowered to negotiate with commissioners to deliver their evaluation products in the most useful format(s) to influence take-up of findings and recommendations.

The final Community of Practice session in the pilot series for members will take place on June 8th 1300 -1400 hrs (BST) and will focus on ‘Evaluation Terms of Reference: Towards Better Practice’.

Also see this website for the upcoming online training course: Handling Complexity module 3 Commissioning and managing
a complex evaluation on July 18th . Open to all.

isabellagdipaolo@gmail.com