Blog from Bridget Dillon Commissionership – Community of Practice 3

The third Community of Practice in The Society’s pilot series on ‘Commissionership – towards better practice’ focused on governing for change during an evaluation, took place on March 23rd.

Discussion highlighted that there is much more said and understood about the various roles which evaluators play (eg facilitator, interviewer, report writer, change agent, task master, analyst, synthesizer…) but much less said and understood about the range of ‘hats’ a commissioner may wear within their role. (eg ToR designer, contractor, risk manager, scrutineer, pay master, user of evaluation …)

It was also striking that the critical importance of building good professional relationships between commissioner and evaluator/ evaluation team, and frontloading robust preparation for the evaluation mentioned in the previous two sessions, was raised again this session.

Useful pointers in relation to governing for change in an evaluation included:

• Setting out expectations of all parties during implementation of the evaluation, at the outset.

Clarity and transparency of roles and responsibilities of commissioner, evaluation manager, evaluator, steering group, communications between parties, contract manager, delivery and payment schedule etc is essential in order to establish a framework of understanding on which to run and build the professional relationship between all those involved, and crucially, between commissioner and evaluator.  Commissioners and evaluators should set this up at first encounter. Evaluators – if this is not forthcoming, ask.

Clarity and transparency as per above, also contributes to smooth continuity in the event – which is often the case – individuals are replaced in their posts during the course of the evaluation. The overall aim is to build a joint relationship which can withstand change and which is ready to handle the consequent adjustments which need to be made.

Evaluations involve a range of actors who are not necessarily directly engaged in the particular evaluation themselves. We could call them ‘hidden actors,’ such as more senior staff who are key drivers for the evaluation behind the scenes. The framework of understanding to run the evaluation and govern the relationship needs to recognise that changes may come from both expected and unexpected sources, and thus should be established so as to be able to govern for ‘no surprises’.

Experience, and indeed common sense, shows that change should be expected to happen as the norm in the course of an evaluation, given there are so many variables involved (personnel, environment, passage of time etc…), Thinking about this, it is extraordinary that a) many evaluations are not considered or contracted with (inevitable) change in mind b) contracts are held to rigidly in some instances, to the extent they limit or undermine what might actually be best for the evaluation.

• Acknowledging when, by whom, and for what specific purpose scrutiny will occur during the evaluation process

Moments of scrutiny are an intrinsic part of the evaluation process. Aspects of scrutiny may apply to quality control of content – scope focus, analysis…, ensuring that the evaluation fits within the parameters set (eg output, style, tone, length,…) ensuring that the evaluation remains fit for purpose, and so forth. These, potentially, are among the most difficult moments in the evaluation process, as commissioner and evaluator may need to deal with disagreement.

Practical suggestion: in line with a ‘no surprises’ approach, provide clarity from the outset who will undertake what level of scrutiny, when, and what it will cover, in the framework which governs the evaluation implementation.

In the past it was often the case that commissioners met with the evaluator/evaluation team at the start of an evaluation and essentially handed it over to them, with what might generously be described as ‘loose scrutiny,’ and met up again over the draft report.

Whilst this remains the experience of some, it is less prevalent practice these days. This is precisely because this way of doing business – ‘commissioning at distance’ – does not guarantee to generate the desired information or product; it is not value for money. It is far too late to deal with problematic issues towards the end of an evaluation. This provides further rationale, if required, for establishing a clear framework to govern ways of working from the outset, and adjust it together and as necessary, as implementation of the evaluation develops.

• Governing for robust ethical practice

Ethical practice in evaluation needs to be a core part of the framework governing an evaluation; ethical practice is not an add-on. This can relate to a wide range of issues including safeguarding of interviewees (eg children, those under threat of persecution, confidentiality…). Discussion in the session focused on data security and GDPR. Mention was made of data protection impact assessments (DPIAs) which those working in public bodies have to provide in advance of an evaluation. See the Information Commissioner’s website for DPIA information and template. https://ico.org.uk/ How do we do DPIAs? Those who have used DPIAs commented that they are ‘organic’ documents and very useful for reference for all parties during an evaluation.

It was suggested that all involved in evaluation should be encouraged to upskill on GDPR – not necessary to expert level – but just to be aware, and be attuned to some of the risks associated with data handling. Given growth of data available, and growing number and ease of channels of communication and reference, this is and will remain an area of significant importance.

• Governing for substantive change during an evaluation

Changes in demands, and/or direction during an evaluation require clear handling. Changes may be changes in policy which could require a course correction, or even a re- coping and involve new evaluation design and tools. It is important to understand where the drive for changes comes from. Is it latter day realization by the commissioners? Is it political pressure? Has the programme been adjusted to incorporate new elements, with evaluation implications for new stakeholder groups, or does it require deeper change in the overall evaluation design or for example, the survey design?

The discussion highlighted that, for both commissioners and evaluators, change is not something to be automatically resisted. As already mentioned – some change is to be expected during the course of an evaluation. Indeed, change can be positive!

‘Scope creep’ is an issue and responsibility for all parties to keep an eye on. Commissioners need to check that they are not asking evaluators to do more than set out in the terms of reference. Equally, evaluators need to signal to commissioners when they consider they are being asked to go beyond the ToR requirements. This is where a good professional relationship is so important. Sticking rigidly to the letter, and giving reasonable leeway are, and feel, very different, and each practice has implications for both parties in the commissioner-evaluator relationship.

‘Creep’ or incremental change can be tricky as it initially may manifest itself as little change, easy to incorporate, but in fact this may provide false sense of security as a particular change may be the tip which indicates that a much bigger/wider change is actually required. This becomes clear when further and further changes are demanded, requiring, over time, substantial adjustment on the part of the evaluators, impacting also their skills (potentially) and effort/time. So, beware incremental ‘creep’. It is also important to recognise that incremental ‘creep’ is not only about specific changes or adjustments, these changes also impact the expectations of parties about the overall evaluation.

Practical suggestion: maintain regular moments of review between commissioner and evaluator, scheduled in the way the implementation of the evaluation is governed, from the outset. Be open with each other on not only what specific change is necessary, but also why and what the implications are, and for whom.

Practical suggestion: establish a ‘developmental contract’ for the evaluation. More than the break clauses which many contracts already incorporate, this would also incorporate time for reflection for those engaged, written into the ITT, and perhaps funds attached to where/if change needs to be instituted. This type of contract needs to be negotiated with legal or procurement departments in the case of large organisations. The advantage of this approach and type of contract enables the commissioner and evaluator to
tailor the evaluation to align the contract better with reality on the ground, as the evaluation develops.

Practical suggestion: distinguishing a role and person in charge of contract management from a role/person in charge of the substance/content of the evaluation. Perhaps this is only possible in large organisations with capacity, and to many this may appear a luxury. However, if possible to do, it can work very well where such a system is well maintained and embedded.

However, it should not be understood that separation of these roles will necessarily work better. Communication between all parties is critical. The more parties added to the issue, the more complicated it will become. However, evaluators much prefer to work to a role which has authority and knowledge about the intervention/evaluand, as opposed to a role focused on contract management.

Practical suggestion: where an evaluator finds themselves in such a position, it is worth actively seeking to establish links with a key player on the (content) intervention side.

Key reflections

Build a sound professional relationship between commissioner and evaluator from the outset for example:

• Establish a set of shared assumptions and choices concerning the evaluation between commissioner and evaluator.
• Establish a clear framework of roles responsibilities of all parties, time-frames, actions etc, for governing and managing the evaluation process.
• Take a pro-active stance – commissioners and evaluators – building on your past experience to set a path going forward which will work; do not wait until things go awry.

See you at the next (Members only) Community of Practice on ‘Commissionership:’
‘Appropriateness and proportionality in an Evaluation – towards better practice’.
Register now for April 19 th 1300 – 1400 BST