TY - JOUR

T1 - Case study meta-analysis in the social sciences. Insights on data quality and reliability from a large-N case survey

AU - Jager, Nicolas W.

AU - Newig, Jens

AU - Challies, Edward

AU - Kochskämper, Elisa

AU - von Wehrden, Henrik

N1 - Publisher Copyright: © 2021 The Authors. Research Synthesis Methods published by John Wiley & Sons Ltd.

PY - 2022/1/1

Y1 - 2022/1/1

N2 - Meta-analytical methods face particular challenges in research fields such as social and political research, where studies often rest primarily on qualitative and case study research. In such contexts, where research findings are less standardized and amenable to structured synthesis, the case survey method has been proposed as a means of data generation and analysis. The method offers a meta-analytical tool to synthesize larger numbers of qualitative case studies, yielding data amenable to large-N analysis. However, resulting data is prone to specific threats to validity, including biases due to publication type, rater behaviour, and variable characteristics, which researchers need to be aware of. While these biases are well known in theory, and typically explored for primary research, their prevalence in case survey meta-analyses remains relatively unexplored. We draw on a case survey of 305 published qualitative case studies of public environmental decision-making, and systematically analyze these biases in the resultant data. Our findings indicate that case surveys can deliver high-quality and reliable results. However, we also find that these biases do indeed occur, albeit to a small degree or under specific conditions of complexity. We identify a number of design choices to mitigate biases that may threaten validity in case survey meta-analysis. Our findings are of importance to those using the case survey method – and to those who might apply insights derived by this method to inform policy and practice.

AB - Meta-analytical methods face particular challenges in research fields such as social and political research, where studies often rest primarily on qualitative and case study research. In such contexts, where research findings are less standardized and amenable to structured synthesis, the case survey method has been proposed as a means of data generation and analysis. The method offers a meta-analytical tool to synthesize larger numbers of qualitative case studies, yielding data amenable to large-N analysis. However, resulting data is prone to specific threats to validity, including biases due to publication type, rater behaviour, and variable characteristics, which researchers need to be aware of. While these biases are well known in theory, and typically explored for primary research, their prevalence in case survey meta-analyses remains relatively unexplored. We draw on a case survey of 305 published qualitative case studies of public environmental decision-making, and systematically analyze these biases in the resultant data. Our findings indicate that case surveys can deliver high-quality and reliable results. However, we also find that these biases do indeed occur, albeit to a small degree or under specific conditions of complexity. We identify a number of design choices to mitigate biases that may threaten validity in case survey meta-analysis. Our findings are of importance to those using the case survey method – and to those who might apply insights derived by this method to inform policy and practice.

KW - Environmental Governance

KW - case survey method

KW - evidence-based governance

KW - inter-rater reliability

KW - meta-analysis

KW - publication bias

KW - case survey method

KW - evidence-based governance

KW - inter-rater reliability

KW - meta-analysis

KW - publication bias

UR - http://www.scopus.com/inward/record.url?scp=85113210278&partnerID=8YFLogxK

UR - https://www.mendeley.com/catalogue/1f30dd8b-f05e-30c5-868c-c6789af4a7f9/

U2 - 10.1002/jrsm.1514

DO - 10.1002/jrsm.1514

M3 - Journal articles

C2 - 34318609

VL - 13

SP - 12

EP - 27

JO - Research Synthesis Methods

JF - Research Synthesis Methods

SN - 1759-2879

IS - 1

ER -