TOWARDS A CORE SET OF COMPETENCIES FOR EVALUATION FOR UKES
Consultation Document (Link to Questionnaire)
Focus and Framework
This consultation document sets out a proposal for the identification by UKES of a core set of competencies that evaluators, managers and commissioners may use to improve the commissioning and conduct of evaluations and evaluation training. It builds on and extends the work of the UKES Guidelines for Good Practice in Evaluation which the UKES Council endorsed in 2002 .
The UKES Guidelines speak to the quality of the commissioning and conduct of evaluation and the reciprocal roles and responsibilities of the different partners in the evaluation enterprise. What they do not address directly are the specific competencies that evaluators need to achieve quality evaluation. It is timely now to consider what these are and how they might be utilized effectively to further the development and professionalization of evaluation.
Other evaluation societies are moving in this general direction but with different emphases. The Canadian Evaluation Society (CES) has proposed a set of competencies focused on certification needs. The DeGEval .has generated a framework of competencies for use in the design of evaluation training programs. The European Evaluation Society launched an initiative in 2008 to examine whether it can reach a core set of competencies across Europe . The International Development Association (IDEAS) is also exploring the issue.
The purpose of this document is (i) to seek the views of UKES members about the rationale and scope of an initiative on core evaluation competencies (Appendix 1) and (ii) to gauge their reactions to a preliminary framework of evaluation competencies (Appendix 2).
Rationale – Why Competencies
For quite some years now evaluation societies have promoted ethical guidelines/ principles/standards/codes to enhance the professionalization of evaluation. Different societies have taken different emphases, some focusing more on the product, others on the process. Not all address the ethics of practice and few have insisted that signing up to a code of practice is a requirement for membership that carries sanctions if the code is breached. With the exception of the UKES Guidelines, most are written with the evaluator in mind.
What few of them address, beyond noting that evaluators should have the knowledge and skills to do the job are the specific competencies evaluators should have which would help assure the quality of evaluation in practice. In this sense competencies go beyond having the knowledge and skills, which are essential to do the job, to include the abilities and qualities evaluators need to use these skills appropriately in the specific evaluation context and to sensitively negotiate the evaluation task.
The quality of evaluation and evaluators is likely to come under greater scrutiny in the coming years as accountability demands continue to increase the role of evaluation in the policy formation process. Commissioners, managers, and participants in evaluation are now more knowledgeable about what evaluation services can offer and more discerning about what counts as quality evaluation and who can do it well. The guidelines that many evaluation societies have issued in the past fifteen years have helped to this end. But a more specific competency framework would extend this work and enhance the professionalization of evaluation.
Risks and rewards
Not all evaluators might agree that formulating a set of core competencies is necessary or desirable to ensure quality evaluation. Some fear that it might provide a stranglehold on what evaluators can do; that it could not cover the variety of competencies needed for different evaluations; and that it might provide commissioners and managers of an evaluation with an in flexible list of competencies that would hold evaluators to account in unhelpful ways.
Looked at more positively and with an intention by all partners in evaluation to enhance the professionalization of evaluation, a framework of competencies could:
- help recently appointed and mid-career evaluators enhance their own professional skills;
- remind experienced evaluators of the need for continuing professional development
- guide commissioners in their selection of evaluators;
- assure participants and managers that the process will be conducted sensitively and well;
- convince stakeholders and audiences of evaluation that the outcome will be a credible, quality evaluation;
- provide a basis for training of new evaluators.
Above all, competencies would help in advancing the recognition of evaluation as a social practice by reaching beyond the mere statement of guidelines, principles or standards to define what is actually required to be competent in the field.
The Scope of Evaluation Competencies for UKES
The UKES Guidelines for Good Practice in Evaluation are perhaps unique in presenting a role for all partners in the evaluation enterprise – evaluator, commissioner, participant. They also indicate how institutions conducting their own evaluation can establish a set of ethical practices in evaluating their own institutions. The guidelines have been in use now for six years and have been welcomed both by members and other evaluation societies seeking to establish ethical guidelines. Members from UKES and other societies have commented on their user -friendly nature and specifically on the roles and obligations suggested for the different groups engaged in the task. These characteristics may be something UKES wishes to emulate in the development of evaluation competencies.
Appendix 1 is a short questionnaire to UKES members to explore the virtue of establishing a core set of competencies for evaluations and to ascertain for what group/s these might be designed. (Link to Questionnaire)
Appendix 2 lists the competencies suggested for consideration by members. These are divided into three sections (i) evaluation knowledge; (ii) professional practice; and (iii) qualities and dispositions.
Evaluation knowledge refers not only to the knowledge base of philosophy and methods we bring to evaluation whether from the social sciences, related disciplines and the professions but also to the knowledge that has arisen within the specific field of evaluation. This includes knowledge of evaluation theories, methodological approaches and their political implications, how we handle our and others’ values and interests in the program and process, and how to report the results of evaluations.
Professional practice refers to the fact that many of the competencies we need to ensure a quality evaluation are born of experience, of having to face difficulties in the field; of having to change tack in a highly politicized evaluation context, for example, where the conditions to effectively implement a well-designed evaluation do not exist; of having to manage mutually conflicting interests and demand for different kinds of data. In short, a good evaluator needs interpersonal and political skills. These skills are hard won. While knowledge of how to conduct evaluation is essential, quality evaluation depends equally on how one acts in the field.
Qualities and dispositions refers to the personal characteristics that enable evaluators to function in difficult circumstances, not to be thrown off task, to maintain an impartial, independent stance, to manage complexity and to tolerate ambiguity and dissonance. Over and above knowledge and the skills acquired in practice, good evaluators need particular attributes and qualities that enable them to conduct a credible evaluation and be recognized as a competent evaluator. It may not be possible to identify all such personal attributes (and they may differ from person to person and in different contexts), but a competencies framework would be incomplete without them.
Use of Competencies
It is not the intention of this first foray into specifying competencies for UKES evaluators and evaluations to take a particular position about their use as other evaluation societies have done. The Canadian Evaluation Society (CES) for instance are working towards accreditation of evaluators; DGEVal are more interested in the training potential. At this stage, we are simply seeking to identify what we think the competencies are that make for good evaluators and with a range of potential purposes in mind on which we also seek your advice. We also hope that our debate on this issue will contribute to the EES initiative and the wider intercountry debate on what core competencies for good evaluation are and how they could be sensitively used to develop the social practice of evaluation.
One final point on which we seek your advice is whether we should at some point aim to have a section on competencies for commissioners and managers of evaluation much as we have a section for different stakeholders in the Guidelines for Good Practice in Evaluation. Do please let us have your views on this as well. The boxes in the questionnaire are all expandable so please comment as fully as you can. We thank you in advance for your comments on this initiative and your help in formulating what would be a useful set of competencies to promote for evaluators and evaluation in the UKES.
(Professor) Helen Simons
On behalf of the UKES Council
May 2009
Appendix 1: Link to Questionnaire
Appendix 2: List of Competencies
Appendix 2
1.1 Appreciates the social and political role played by evaluation
1.11 Shows awareness of evaluation history and knowledge
1.12 Appreciates the linkages & differences between evaluation & social research
1.13 Understands program theory and its implications
1.14 Can relate evaluation to policy environments
1.15 Demonstrates familiarity with theories of causality and attribution
1.16 Shows awareness of different modes of generalization
1.2 Understands evaluation methods and approaches
1.21 Uses appropriate evaluation concepts and terms
1.22 Knows how to design and structure an evaluation
1.23 Comprehends the value of diverse evaluation approaches
1.24 Demonstrates appropriate use of these in specific contexts
1.25 Shows familiarity with quantitative & qualitative methodologies and their underlying philosophical assumptions
1.26 Grasps the potential and limits of quantitative methods (including randomized experiments, cost benefit analysis, survey); & qualitative methods (including case study, interview & observation)
1.3 Shows ability to use evaluation tools of data collection and analysis to:
1.31 Design case studies, surveys, interviews, observations
1.32 Set up and use evaluation monitoring systems
1.33 Determine appropriate indicators & questions for evaluation task
1.34 Analyze and interpret data with variety of strategies
1.35 Establish criteria for validity and for ‘valuing’ the evaluation
1.36 Report fairly and justly to agreed audiences in the public interest
2. Professional practice competencies
2.1 Demonstrates capacity to deliver evaluations
2.11 Ascertains the evaluative context
2.12 Appraises policy logic and program theory
2.13 Selects appropriate approaches, methods and tools
2.14 Chooses judicious and relevant evaluation criteria
2.15 Knows when to involve evaluators with different skills
2.16 Demonstrates effective management of the evaluation
2.17 Makes sensitive ethical decisions in practice
2.18 Interprets evaluation results in context
2.2 Demonstrates interpersonal skills
2.21 Writes fluently and communicates clearly
2.22 Able to work independently and in a team
2.23 Uses sound and sensitive negotiating skills
2.24 Demonstrates cultural/gender sensitivity
2.25 Nurtures professional relationships
3 Qualities and disposition competencies
3.1 Upholds ethical and democratic values
3.2 Respects clients, stakeholders and participants
3.3 Is responsive and flexible in changing context
3.4 Exercises sound, rigorous and fair judgment
3.5 Is able to handle conflict and remain impartial
3.6 Displays self-knowledge and pursues professional development