Case study meta-analysis in the social sciences. Insights on data quality and reliability from a large-N case survey

Research output: Journal contributionsJournal articlesResearchpeer-review

Authors

Meta-analytical methods face particular challenges in research fields such as social and political research, where studies often rest primarily on qualitative and case study research. In such contexts, where research findings are less standardized and amenable to structured synthesis, the case survey method has been proposed as a means of data generation and analysis. The method offers a meta-analytical tool to synthesize larger numbers of qualitative case studies, yielding data amenable to large-N analysis. However, resulting data is prone to specific threats to validity, including biases due to publication type, rater behaviour, and variable characteristics, which researchers need to be aware of. While these biases are well known in theory, and typically explored for primary research, their prevalence in case survey meta-analyses remains relatively unexplored. We draw on a case survey of 305 published qualitative case studies of public environmental decision-making, and systematically analyze these biases in the resultant data. Our findings indicate that case surveys can deliver high-quality and reliable results. However, we also find that these biases do indeed occur, albeit to a small degree or under specific conditions of complexity. We identify a number of design choices to mitigate biases that may threaten validity in case survey meta-analysis. Our findings are of importance to those using the case survey method – and to those who might apply insights derived by this method to inform policy and practice.
Original languageEnglish
JournalResearch Synthesis Methods
Volume13
Issue number1
Pages (from-to)12-27
Number of pages16
ISSN1759-2879
DOIs
Publication statusPublished - 01.01.2022

Bibliographical note

Publisher Copyright:
© 2021 The Authors. Research Synthesis Methods published by John Wiley & Sons Ltd.

    Research areas

  • Environmental Governance - case survey method, evidence-based governance, inter-rater reliability, meta-analysis, publication bias

Documents

DOI

Recently viewed

Publications

  1. Das neue AGG
  2. 3D Simulation of Electric Arcing and Pressure increase in an Automotive HVDC Relay During a Short Circuit Situation
  3. Diffusion of the Balanced Scorecard
  4. Is the market classification of risk always efficient?
  5. Governmental activity, integration, and agglomeration
  6. Daily breath-based mindfulness exercises in a randomized controlled trial improve primary school children’s performance in arithmetic
  7. Use of Chemotaxonomy To Study the Influence of Benzalkonium Chloride on Bacterial Populations in Biodegradation Testing
  8. Digital game culture(s) as prototype(s) of mediatization and commercialization of society
  9. Oxygen dependence in the photoreaction of the pesticide metamitron
  10. Launching insectphylo.org; a new hub facilitating construction and use of synthesis molecular phylogenies of insects
  11. Bunker schreiben
  12. Credit constraints and exports: A survey of empirical studies using firm level data
  13. Exports, R&D and Productivity
  14. Towards a Heuristic for Scheduling Offshore Installation Processes
  15. What is a Digital Object?
  16. Altersetin, a new antibiotic from cultures of endophytic Alternaria spp. taxonomy, fermentation, isolation, structure elucidation and biological activities
  17. The influence of vertical integration and property rights on network access charges in the German electricity market
  18. Sustainability-oriented technology exploration: managerial values, ambidextrous design, and separation drift
  19. Geometrical Characterization of Polyethylene Oxide Nanofibers by Atom Force Microscope and Confocal Laser Scanning Microscope
  20. Groundwater intrusion into leaky sewer systems
  21. Battery as a mediating technology of organization
  22. Development and validation of a stability-indicating RP-HPLC method for the determination of paracetamol with dantrolene or/and cetirizine and pseudoephedrine in two pharmaceutical dosage forms
  23. Remaining time and opportunities at work: Relationships between age, work characteristics, and occupational future time perspective
  24. Reconfigurable Control System for Plants with Variable Structure
  25. Optimal scheduling for Automated Guided Vehicles (AGV) in blocking job-shops
  26. Christian Steinbacher
  27. A black box identification in frequency domain
  28. Can Geodesign Be Used to Facilitate Boundary Management for Planning and Implementation of Nature-based Solutions?
  29. Citizen Action in the Time of the Network
  30. Implementierung und langfristige Wirkungen des Projekts ‚Jedem Kind ein Instrument‘.
  31. The New Media
  32. Biodiversity across trophic levels drives multifunctionality in highly diverse forests