17 March 2021
UK Evaluation Journal Club – Discussion with author
Dr Barbara Befani will be joining the Journal Club for a second time, this time to discuss her paper “Diagnostic evaluation and Bayesian Updating: Practical solutions to common problems”
This paper is a companion paper to ‘Quality of quality: A diagnostic approach to qualitative evaluation’ published in Evaluation in 2020 [doi:10.1177/1356389019898223 ] that provides a conceptual introduction to diagnostic evaluation and Bayesian Updating as a general variation of Theory-Based Evaluation. Please refer to this paper for more general or theoretical information about the approach.
Dr Barbara Befani is a Research Fellow at the University of Surrey Centre for Research in Social Simulation and a Research Associate at the University of East Anglia.
Taking part in a Journal Club is very similar to taking part in a Book Club but can be organised in different ways suited to the material being discussed. Just like a book club you are asked to read a journal article in advance so you can discuss it with others. The person who choose the article, or in this case the author, usually provides an overview of the methodological approach or findings described in the paper, before opening up the discussion to the group. Journal clubs can involve group members critically reviewing an article using critical appraisal tools particularly when the aim is to develop skills in evidence synthesis. Or they can provide an opportunity for group members to discuss the applicability and adaptability of a proposed methodology to their area of research and evaluation.
This journal club meeting provides an opportunity to discuss with Barbara :
- Basic tenets of “diagnostic (theory-based evaluation”
- Why we can apply Process Tracing and Bayesian Updating principles to Theory-Based Evaluation in general
- Why we “should” do the above ?
- What is required in practical terms ?
- How you can quickly and inexpensively improve your way of applying Theory-Based Evaluation
The journal club is open to members of the Society and the number of places are limited to allow attendees to contribute and feel part of the discussion.
This article discusses several practical issues arising with the application of diagnostic principles to theory-based evaluation (e.g. with Process Tracing and Bayesian Updating). It is structured around three iterative application steps, focusing mostly on the third. While covering different ways evaluators fall victims to confirmation bias and conservatism, the article includes suggestions on which theories can be tested, what kind of empirical material can act as evidence and how to estimate the Bayes formula values/update confidence, including when working with ranges and qualitative confidence descriptors. The article tackles evidence packages (one of the most problematical practical issues), proposing ways to (a) set boundaries of single observations that can be considered independent and handled numerically; (b) handle evidence packages when numerical probability estimates are not available. Some concepts are exemplified using a policy influence process where an institution’s strategy has been influenced by a knowledge product by another organisation.
Befani B. Diagnostic evaluation and Bayesian Updating: Practical solutions to common problems. Evaluation. 2020;26(4):499-515. doi:10.1177/1356389020958213
The paper is open access and can be downloaded without a subscription to Evaluation journal.
This event is for members only, who can register here to join us