• Guidelines
• Courses
• Journals
• Evaluation Societies
• Online Resources
• Glossary
• Bibliography

 Public Library/Glossary

updated 18/02/03

Glossary of Evaluation Terms

Source: Evaluating EU Expenditure Programmes: A Guide to Intermediate and Ex Post Evaluation. Appendix 1.

Analysis of variance

A widely-used statistical inference technique, based on comparing the variance between samples with the variance within samples. This can tell us whether there is any systematic difference between samples that needs to be explained. See also sample, statistical analysis, variance.

Analytical agenda

A logical structure imposed on the different questions to be asked in an evaluation. This serves to transform the general, often vague, questions which those requesting the evaluation have in mind into questions which are precise enough to be manageable by evaluation research methods. Once the analytical agenda has been drawn up, those responsible for commissioning the evaluation have to ask whether the intervention is indeed evaluable in terms of this analytical agenda. See also evaluability, evaluability assessment, evaluation project, intervention logic.

Audit

A control function, which is primarily concerned with verifying the legality and regularity of the implementation of resources in a programme. Audit has traditionally covered areas such as the verification of financial records (financial audit). See also performance audit, evaluation.

Before-and-after design

An example of a quasi-experimental design in which one simply compares the relevant state of the world after the intervention with its state beforehand and attributes any difference to the effects of the intervention. A particular weakness of this design is the possibility that something else besides the intervention accounts for all or part of the observed difference over time. See also control group, counterfactual situation, evaluation design, internal validity, intervention logic, quasi-experimental designs, programme group.

Benchmarks

Standards by which the performance of an intervention can be assessed in a non-arbitrary fashion. An obvious way of deriving benchmarks would be to examine the intervention's objectives as expressed by expected outputs, results and outcomes. Ideally, benchmarks should allow us to compare the performance of an intervention with that of other policy instruments in the same field of action or in a related one. See also general objectives, indicator, intervention, objectives, operational objectives, outcomes, outputs, results, specific objectives.

Case studies

A data collection technique involving the examination of a limited number of specific cases or projects which the evaluator anticipates will be revealing about the programme as a whole. Case studies tend to be appropriate where it is extremely difficult to choose a sample large enough to be statistically generalisable to the population as a whole; where generalization is not important; where in-depth, usually descriptive data is required; and where the cases or projects to be studied are likely to be quite complex. See also case study designs, data collection.

Case study designs

A class of evaluation designs in the descriptive rather than the causal approach. It is often the case that an evaluation design will be based on an in-depth study of one or more specific cases or situations. See also case studies, evaluation design.

Chatty bias

A general problem which arises when the views of more outspoken individuals (e.g. experts) tend to stand out, although their views may not be representative. See also expert opinion.

Comparative change design

An example of a quasi-experimental design in which any known or recognisable difference between the programme and control groups is taken into account in the statistical analysis. The problems with this design are, firstly, that there may be some other factor which explains some or all of the variation in the intervention and in the observed effects, and, secondly, that there may be initial differences between the programme and control groups which have an influence on observed effects and which can therefore become confounded with the influence of the programme on these effects (selection bias). See also control group, counterfactual situation, evaluation design, internal validity, intervention logic, quasi-experimental designs, programme group, selection bias.

Control group

A group of subjects which have not been exposed to an intervention. The control group should resemble the programme group (the subjects which have been exposed to the intervention), so that systematic differences between the two groups may be attributed to the effects of the intervention once other plausible alternative hypotheses have been eliminated or discounted. See also counterfactual situation, evaluation design, intervention logic, programme group.

Cost-benefit analysis

A judgemental technique in which a researcher compares all social and private costs and benefits of a programme with a view to determining whether the benefits exceed the costs, and if so by how much. Social costs and social benefits usually have to be measured by some indirect means and converted into monetary values so that a comparison can be made with private costs and benefits. Furthermore, it may not be appropriate to use prevailing market prices. Consider a situation of very high unemployment. In this case, the real cost of labour may be much lower than the prevailing market wage. The opportunity cost (the next best use of the otherwise unemployed workers had the project not gone ahead) is lower than the prevailing wage rate, and this low opportunity cost has to be represented by a shadow price which has to be derived somehow. See also cost-effectiveness analysis.

Cost-effectiveness analysis

A judgmental technique in which the researcher quantifies the costs and benefits associated with a programme on the basis of the same principles which apply to cost-benefit analysis, but there is no requirement to transfer benefits into common monetary units. See also cost-benefit analysis, effectiveness.

Counterfactual situation

The situation which would have arisen had the intervention not taken place. In order to derive the counterfactual situation we need an evaluation design. Except for the theoretical case of the ideal experimental design, we can never know the counterfactual situation with certainty. Real world evaluation designs tend to be based on an estimate of the counterfactual derived either from comparing subjects who were exposed to an intervention with a comparison group who were not exposed, or from examining subjects before and after exposure. See also control group, evaluation design, ideal experimental design, intervention logic, programme group.

Top of the Page

Criterion-population design

An example of a quasi-experimental design, which attempts to improve on the comparative change design. In the latter, the programme and control groups are two distinct groups drawn from a hypothetical larger population. In the criterion-population design, however, the hypothetical population is identified and used for the comparison group. In this case, the possibility of selection bias is confined to just one group - the programme group. This design is particularly appropriate where the evaluator cannot easily create a control group but does have access to information about the larger population from which the programme group is drawn. See also control group, comparative change design, counterfactual situation, evaluation design, internal validity, intervention logic, quasi-experimental designs, programme group, selection bias.

Data

Known facts which can be used as a basis for inference. Subjective data involve personal feelings, attitudes and perceptions; objective data relate to observable facts. Quantitative data involve numerical observations; qualitative data are non-numerical and related to categories. Longitudinal data are collected over time; cross-sectional data are collected from the same point in time, but from a variety of different geographical areas, etc. Primary data are taken directly from original sources or collected first hand; secondary data have undergone extensive manipulation and interpretation. See also data analysis, data collection.

Data analysis

The main techniques used to interpret information about an intervention for use in an evaluation are statistical analysis, the use of models, non-statistical analysis and judgement techniques, such as cost-benefit analysis, cost-effectiveness analysis and multi-criteria analysis. See also cost-benefit analysis, cost-effectiveness analysis, data collection, models, multi-criteria analysis, non-statistical analysis, statistical analysis.

Data collection

The main techniques used to gather information about an intervention for use in an evaluation are surveys, case studies, natural observations, expert opinion, reviews of programme documents and literature reviews. See also case studies, data analysis, evaluation design, expert opinion, literature reviews, natural observations, programme document reviews, surveys.

Deadweight

Deadweight is defined as effects which would have arisen even if the intervention had not taken place. Deadweight usually arises as a result of inadequate delivery mechanisms which fail to target the intervention's intended beneficiaries sufficiently well. As a result, other individuals and groups who are not included in the target population end up as recipients of benefits produced by the intervention. Deadweight is really a special case of programme inefficiency. See also delivery mechanisms, efficiency, target population.

Delivery mechanisms

The organisational arrangements which provide the goods and services funded by the intervention to its intended beneficiaries, i.e. its target population. See also target population.

Delphi technique

A technique which can be used to systematise expert opinions. Experts are consulted separately in a number of different rounds. In each successive round, each individual is told the views of the other experts in the previous round. This technique can be used to arrive at a consensus, or at least to reduce disagreements. See also Abacus of Régnier, expert opinion.

Top of the Page

Displacement

Displacement and substitution are two closely related terms which are used to describe situations where the effects of an intervention on a particular individual, group or area are only realised at the extent of other individuals, groups or areas. Consider, for example, the case of a programme to provide employment subsidies. In a firm which benefits from this programme, subsidised workers may take the place of unsubsidised workers who would otherwise have been employed by that firm. This is known as substitution. Alternatively, a firm benefiting from the employment subsidies may win business from other firms which do not participate in the scheme. Thus, the jobs created in the participating firm may be partly or wholly offset by job losses in other firms. This is known as displacement.

Dissemination

The set of activities by which knowledge about an evaluation is made available to the world at large. See also reporting.

Double-loop learning

A type of feedback, in which the information compiled by an evaluation is used to call into question the very existence of an intervention or to bring about major changes in its basic orientations. Double-loop learning is almost always the result of summative evaluations. It is of key importance in focusing the activities of the European Union towards meeting the evolving needs of its citizens. See also feedback, formative evaluation, single-loop learning, summative evaluation.

Effectiveness

To what extent have the intervention's impacts contributed to achieving its specific and general objectives? See also cost-effectiveness analysis, general objectives, impacts, intervention logic, objectives, outcomes, results, specific objectives.

Efficiency

How economically have an intervention's inputs been converted into outputs and results? See also inputs, intervention logic, outputs, results.

Evaluability

The issue of whether or not the questions raised by a given analytical agenda for an evaluation are at all answerable by an evaluator using appropriate research methods. To know whether the questions can be answered with an acceptable degree of credibility, it is often advisable to perform an evaluability assessment. If an intervention is not evaluable in terms of this analytical agenda (e.g. because adequate data are not yet available), this can lead to a decision to postpone the evaluation or to draw up a new, more realistic analytical agenda. See also analytical agenda, evaluability assessment, evaluation project.

Evaluability assessment

An attempt to determine whether or not the questions raised by a given analytical agenda for an evaluation are at all answerable by an evaluator using appropriate research methods. See also analytical agenda, evaluability, evaluation project.

Evaluation

An in-depth study which takes place at a discrete point in time, and in which recognised research procedures are used in a systematic and analytically defensible fashion to form a judgement on the value of an intervention.

Evaluation design

A model which is used to describe an intervention and provide evidence on the effects which may be attributable to it. Evaluation designs are either causal or descriptive in nature. A given design should lead to the choice of one or more data analysis and collection techniques. See also counterfactual situation, data analysis, data collection, ideal experimental design, intervention logic.

Evaluation project

A sequence of logical steps starting out from the formulation of problems and interests motivating the evaluation to arrive at a series of questions that can be addressed in an analytically acceptable way. Typically, the commissioner establishes an evaluation plan that sets out a framework in which the evaluation is to be conducted and then chooses the evaluator.

Evaluation report

The end product of an evaluation, the evaluation report should follow a logical structure and meet the needs of the evaluation sponsors and the principal stakeholders. Evaluation reports should include an executive summary. The structure of the report is usually specified by the sponsors in the terms of reference. See dissemination, evaluation sponsors, executive summary, reporting, stakeholders, terms of reference.

Evaluation sponsors

The department or agency responsible for commissioning the evaluation. See also management structure, organisational structure, stakeholders, steering group, terms of reference.

Ex ante evaluation

An evaluation conducted before the implementation of an intervention. Also referred to as an "appraisal". See also evaluation, ex post evaluation, intermediate evaluation.

Ex post evaluation

An evaluation conducted either on or after completion of an intervention. See also evaluation, ex ante evaluation, intermediate evaluation.

Ex post facto design

An example of a descriptive design, which can be used where the evaluator cannot select who is to be exposed to the programme, and to what degree. These designs have been used to examine interventions with universal coverage. See also control group, counterfactual situation, evaluation design, intervention logic, programme group.

Executive summary

It is likely that only a small proportion of the target audience will read the full evaluation report. It is therefore advisable to produce a well-written executive summary of no more than five pages in length. This summary forms part of the report and can also be distributed as a stand-alone document. See also evaluation report.

Expert opinion

A data collection technique, similar to a survey, which relies on the necessarily subjective views of experts in a particular field. It is not recommended to rely on expert opinion as a sole data source, for example, because of problems with so-called "chatty bias". See also chatty bias, data collection, Delphi technique, surveys.

External evaluation

An evaluation which is performed by persons outside the organisation responsible for the intervention itself. See also evaluation, internal evaluation.

External validity

The confidence one can have about whether or not one's conclusions about the intervention can be generalised to fit circumstances, times, people, and so on, other than those of the intervention itself. A threat to external validity is an objection that the evaluation design does not allow causal inference about the intervention to be generalised to different times, places or subjects to those examined in the evaluation. See also evaluation design, internal validity, intervention, intervention logic.

Feedback

The process by which the information compiled by an evaluation is used by decision-makers to either change the way in which an intervention is implemented, or to bring about a more fundamental change in the basic orientations of the intervention, including calling into question its very existence. See also double-loop learning, single-loop learning.

Formative evaluation

An evaluation concerned with examining ways of improving and enhancing the implementation and management of interventions. Formative evaluations tend to be conducted for the benefit of those managing the intervention with the intention of improving their work. See also evaluation, summative evaluation.

General objectives

The desired effects of an intervention expressed in terms of outcomes, i.e. the longer-term impact of the intervention on society (e.g. to reduce unemployment among the long-term unemployed). See also intervention logic, objectives, operational objectives, outcomes, specific objectives.

Hawthorne effect

The term "Hawthorne effect" is used to explain situations where an experiment cannot be trusted because the very fact that the experiment is taking place is influencing the results obtained. This reminds us that programme staff and beneficiaries can behave quite differently from their normal patterns if they know that they are being observed. See also natural observations.

Ideal experimental design

A theoretical way of deriving the counterfactual situation, and hence the net impact of an intervention. It involves comparing two groups which are identical in all respects except one: exposure to the intervention. Differences between the group which has been exposed (the programme group) and the group which has not (the control group) are then attributable to the intervention. In the real world, this design does not exist since we can never be absolutely certain that the two groups are identical in all other respects. The potential non-equivalence of the two groups weakens the validity of any causal inference about the intervention. A number of real world evaluation designs are available which each have their own strengths and weaknesses. See also control group, counterfactual situation, evaluation design, intervention logic, programme group, quasi-experimental designs, true experimental designs.

Impacts

A general term used to describe the effects of a programme on society. Impacts can be either positive or negative and foreseen or unforeseen. Initial impacts are called results, whilst longer-term impacts are called outcomes. See also outcomes, results.

Indicator

A characteristic or attribute which can be measured to assess an intervention in terms of its outputs or impacts. Output indicators are normally straightforward. Impact indicators may be more difficult to derive, and it is often appropriate to rely on indirect indicators as proxies. Indicators can be either quantitative or qualitative. The term "performance indicators" is also used. See also benchmarks, general objectives, impacts, operational objectives, outputs, specific objectives.

Inputs

The human and financial resources involved in the implementation of an intervention. See also intervention, intervention logic.

Intermediate evaluation

An evaluation conducted during the implementation of an intervention. See also evaluation, ex ante evaluation, ex post evaluation.

Internal evaluation

An evaluation which is performed by members of the organisation responsible for the intervention itself. See also self evaluation, external evaluation.

Internal validity

The confidence one can have in one's conclusions about what the intervention actually did accomplish. A threat to internal validity is an objection that the evaluation design allows the causal link between the intervention and the observed effects to remain uncertain. It may be thought of as a question of the following nature: could not something else besides the intervention account for the difference between the situation after the intervention and the counterfactual? See also counterfactual situation, evaluation design, external validity, intervention, intervention logic, selection bias.

Interrupted time-series design

An example of a quasi-experimental design. It involves obtaining several measurements over time both before and after exposure to a programme in order to create a time series of observations. It is an improvement on the before-and-after design. See also before-and-after design, control group, counterfactual situation, evaluation design, internal validity, intervention logic, quasi-experimental designs, programme group.

Intervention

A generic term used to cover all public actions. See also policy, programme, project.

Intervention logic

The conceptual link from an intervention's inputs to the production of its outputs and, subsequently, to its impacts on society in terms of results and outcomes. The examination of the programme's intervention logic will be of central importance in most evaluations. The evaluator needs to ask how the programme achieves its specific objectives, and how do the specific objectives contribute to the attainment of the general objectives? The terms "theory of action", "programme logic" and "programme theory" are sometimes used to mean more or less the same thing. See also general objectives, impacts, inputs, intervention, objectives, operational objectives, outcomes, outputs, results, specific objectives.

Literature reviews

Top of the Page

Management structure

Mean

The most commonly used descriptive statistic, it tells us the average of a set of values. See also standard deviation, statistical analysis.

Models

There are various different models which seek to represent how an intervention changes important socio-economic variables. Such models are normally taken from previous research. The main types of models are: (i) input-output models, which allow a researcher to systematically examine the linkages between the different parts of an economy, as the inputs of one industry can be thought of as the outputs of other industries; (ii) microeconomic models, which are designed to examine the behaviour of households and firms in specific industries and markets using equations which represent the supply and demand functions for a particular good or service; (iii) macroeconomic models, which are used to model the behaviour of the economy as a whole and the evolution of important macroeconomic variables (such as inflation, employment, growth and the trade balance) over time; and, (iv) statistical models, which are frequently used to examine relationships between specific programme effects. See also data analysis, statistical analysis.

Monitoring

The continuous process of examining the delivery of programme outputs to intended beneficiaries, which is carried out during the execution of a programme with the intention of immediately correcting any deviation from operational objectives. Evaluation, on the other hand, is carried out at a discrete point in time, and consists of an in-depth study. Monitoring often generates data which can be used in evaluations. See also evaluation.

Multi-criteria analysis

A decision-making tool which can be adapted to form judgements about interventions. Multi-criteria analysis allows us to formulate judgements on the basis of multiple criteria, which may not have a common scaling and which may differ in relative importance.

Natural observations

A data collection technique in which the evaluator makes on-site visits to locations where the intervention is in operation and directly observes what is happening. Observational data can be used to describe the setting of the intervention, the activities which take place in the setting, the individuals who participate in these activities (who may or may not be aware that they are being observed), and the meaning of these activities to the individuals. This form of data collection is particularly vulnerable to the Hawthorne effect. See also data collection, Hawthorne effect.

Needs

The socio-economic problems which an intervention aims to address, expressed from the point of view of its target population. For example, the need to improve job opportunities for long-term unemployed workers who may suffer from a lack of relevant skills. See also objectives, target population.

Non-statistical analysis

A general term used to describe the analysis of mainly qualitative data which is typically used in conjunction with statistical analysis (of either qualitative or quantitative data). Usually, this includes an assessment of the reliability of any findings derived from such methods. See also data, data analysis, statistical analysis.

Objectives

The desired effects of an intervention. See also general objectives, needs, operational objectives, specific objectives.

Operational objectives

The desired effects of an intervention expressed in terms of outputs, i.e. the goods and services produced by an intervention (e.g. to provide professional training courses to the long-term unemployed). See also general objectives, intervention, intervention logic, objectives, outputs, specific objectives.