The ResQ methodology
The ResQ methodology is a multi-method approach (see Goertz, 2017) that combines realist evaluation science with qualitative comparative analysis. Below we give a short overview of what multimethod research and two constituting methodological approaches are. Finally, we give an extensive overview of how the ResQ methodology works.
In his book Multimethod Research, Causal Mechanisms, and Case Studies, Prof. Gary Goertz puts forward the idea of the research triad. The proposition entails that 'good' multimethod research should cover the three complementary corners of the triad. The latter consists of 'causal mechanisms', 'cross-case inference (i.e. generalization)', and 'within-case inference (i.e. case studies)' (see figure 1) (Goertz 2017). To put it in a nutshell, 'good multimethod research' uses case studies to explore the relevant causal mechanisms after which it it uses statistics or QCA or multiple case studies or experiments to generalize (either statistically or analytically) the found mechanisms across cases.
Figure 1: The research triad (Goertz 2017, p. 2)
It is important not to confuse multi-method research with mixed method research. The latter concerns the use of quantitative and qualitative research methods in one research design. Multimethod research, is not so much concerned with the form of the methods, as with the ultimate purpose of the method. Some methods are used for causal analysis at the level of single cases (e.g. process tracing), whereas other methods are used for cross-case inferences. Combining these methods in a complementary way in one research design, creates a strong approach for theory building around causal mechanisms.
It is within this framework of the multimethod approach that the ResQ study should be situated. As will become clear, we use realist evaluations as secondary sources for the analysis of causal mechanisms. Subsequently, we introduce qualitative comparative analysis (QCA) to strengthen the analytical generalization of the 'demi-regularities'.
Realist evaluation (Pawson & Tilley 1999) is a theory-based approach to evaluation, which means that an evaluation starts with a program theory that explains how the intervention will lead to certain outcomes. This program theory is subsequently tested and evaluated and finally updated (see figure 2).
Figure 2:Realist evaluation cycle; Adapted and simplified from Marchal et al (2012)
What sets realist evaluation apart from other theory-based evaluations approaches (like Theory of Change or process tracing) is that it is founded on the (scientific) realist philosophy of science and focused on “what works for whom, when, where and why” (Pawson, 2013). Therefore, it views causation in a generative way and is organized around generative mechanisms. These are “underlying entities, processes, or structures which operate in particular contexts to generate outcomes of interest” (Astbury & Leeuw, 2010, p. 368).
​
A commonly used analogy, despite it coming from the exact sciences, is the mechanism 'gravity'. When I am holding a ball and I pull away my hand it is the mechanism 'gravity' that causes it to fall to the ground is the underlying causal mechanism of the event of a ball falling to the ground when I let go of the ball. In contrast, the more common successionist approach to causation would say that I caused the ball to drop by letting it go. Hence, realist evaluation is not about understanding which actions (interventions) lead to which outcomes, but about the mechanisms that are triggered causing the observed outcomes.
Another important part of the realist approach is the assumption that mechanisms are ‘demi-regularities’ (in contrast with general laws) that are only triggered in specific contexts to generate specific outcomes. To highlight this specific feature of mechanisms, realist evaluators use a heuristic that explicitly links generative mechanisms with elements from the context and their outcomes: the “context-mechanism-outcome configuration” or CMOC. This heuristic is central to every realist evaluation as it guides the formulation of the hypotheses derived from the program theory and the analysis of the data. Indeed, the golden thread throughout a realist evaluation is the study of the mechanisms that caused the observed outcomes and the context that enabled the triggering of these mechanisms. This finally leads to an updated CMO-configuration and program theory.
​
This aspect of theory building links realist evaluation with research. Moreover, following Sayer (1992), Pawson proposes that these CMOCs are “generic conceptual platforms” (Pawson, 2013, p. 94) that are able to link findings from similar programs in similar contexts yet different domains.[1]
​
[1] For example, incentivization is a mechanism that is triggered in certain interventions in the healthcare sector but also in the education or the agriculture sectors. Lessons on incentivization (for whom, what and when it works) can thus be drawn from these very different domains and contribute to theory-building in general.
Qualitative comparative analysis (QCA) (Ragin, 2014 [1987]) is a set-theoretic method that uses fuzzy and Boolean logic to analyze multiple cases in order to determine the necessary and sufficient conditions for a predefined outcome to occur (Goertz & Starr, 2002; Ragin, 2014 [1987]). Like realism, it is a case-based approach that takes into account the richness of each of the cases included in the analysis. By comparing cases and their outcomes it aims to minimize the conditions deemed relevant for an outcome to occur.
​
Two of the main underlying concepts are 'equifinality' and 'multifinality'. The former means that several pathways exist to achieve the same outcome, while the latter entails that one specific condition can lead to several different outcomes depending on the other conditions in the context. Indeed, the latter claim on configurational causality is similar to the one made by the realist approach on 'demi-regularities'.
​
A QCA analysis starts with the identification of the conditions that are deeemed relevant for a certain outcome to occur. As in the realist approach this can be inspired by theories, insights from key-informants, or experience of the researcher. This is not a final decision as QCA is a iterative process in which the researcher goes back and forth between the different phases.
​
Next a data matrix is created. Data is collected on both the identified conditions and the outcomes for each case. Subsequently, the data is calibrates on the set-theoretic membership of the case in the different conditions and the outcome. [2]
The data matrix is subsequently transformed into a truth table. A truth table consists of 3 elements: a list of every possible combination of conditions, the cases that correspond to this combination of conditions and the outcomes (1 or 0) that correspond to these cases.
​
Finally, this truth table is analyzed using a minimization process which leads to a solution that depicts a combination of sufficient and necessary conditions that are observed at the same time as the outcome.
​
[2]For example, a salary bonus that comprises 5 % of the base salary can correspond to a calibrated score of 0,4 in the set 'high bonus'. That means that it is more out than in the set of 'high bonus'.
The ResQ study tries to take full advantage of these earlier described 'generic conceptual platforms' (or mechanisms), as they enable us to bridge the domains of evaluation and science and transform evaluation results into theories. Indeed, the ResQ approach revolves around the different mechanisms. Each individual evaluation gives us more information about the conditiond under which they are being triggered. By synthesizing all this information we are able to create theories about the different mechanisms that explain when they are triggered. However, we tend to be overhelmed by the conditions that matter.
​
Interestingly, Pawson (2013) and Sayer (1992) put relatively strong emphasis on a need to know the necessary components of and scope conditions for these generic conceptual platforms. However, they are not as clear about how to discover these necessary components and conditions. In order to do so, the ResQ study uses the technique of qualitative comparative analysis (QCA).
After having identified the different mechanisms that are of relevance for the intervention under study (in this case PBF), the ResQ approach searches the peer-reviewed literature for studies that can inform us about the relevant conditions for a mechanism to get triggered. This will lead to a list of conditions of which it is difficult to discern which ones are sufficient, necessary or irrelevant. By applying a QCA on this list of conditions that are linked (or not) to the triggering of a mechanism we are able to create a theory that describes when and where a certain mechanism is triggered. This will give valuable information to policymakers who aim to implement policies that aim to trigger these mechanism or at least the outcomes that these mechanisms cause.
​
.
​