Case study meta-analysis in the social sciences. Insights on data quality and reliability from a large-N case survey

Research output: Journal contributionsJournal articlesResearchpeer-review

Authors

Meta-analytical methods face particular challenges in research fields such as social and political research, where studies often rest primarily on qualitative and case study research. In such contexts, where research findings are less standardized and amenable to structured synthesis, the case survey method has been proposed as a means of data generation and analysis. The method offers a meta-analytical tool to synthesize larger numbers of qualitative case studies, yielding data amenable to large-N analysis. However, resulting data is prone to specific threats to validity, including biases due to publication type, rater behaviour, and variable characteristics, which researchers need to be aware of. While these biases are well known in theory, and typically explored for primary research, their prevalence in case survey meta-analyses remains relatively unexplored. We draw on a case survey of 305 published qualitative case studies of public environmental decision-making, and systematically analyze these biases in the resultant data. Our findings indicate that case surveys can deliver high-quality and reliable results. However, we also find that these biases do indeed occur, albeit to a small degree or under specific conditions of complexity. We identify a number of design choices to mitigate biases that may threaten validity in case survey meta-analysis. Our findings are of importance to those using the case survey method – and to those who might apply insights derived by this method to inform policy and practice.
Original languageEnglish
JournalResearch Synthesis Methods
Volume13
Issue number1
Pages (from-to)12-27
Number of pages16
ISSN1759-2879
DOIs
Publication statusPublished - 01.01.2022

Bibliographical note

Publisher Copyright:
© 2021 The Authors. Research Synthesis Methods published by John Wiley & Sons Ltd.

    Research areas

  • Environmental Governance - case survey method, evidence-based governance, inter-rater reliability, meta-analysis, publication bias

Documents

DOI

Recently viewed

Publications

  1. Doing statistics, enacting the nation
  2. Empathy as a motivator of dyadic helping across group boundaries
  3. Risk management with management control systems
  4. Study harder? the relationship of achievement goals to attitudes and self-reported use of desirable difficulties in self-regulated learning
  5. Broad values as the basis for understanding deliberation about protected area management
  6. Shared mobility business models
  7. Towards a cross-paradigmatic framework of the social acceptance of energy systems
  8. Emotional reactivity and interoceptive sensitivity
  9. A specification schema for software connectors
  10. From niche to mainstream
  11. Digital Games and Fan-Discourse
  12. Technik – Magie – Medium
  13. Utilization of food waste in continuous flow cultures of the heterotrophic microalga Chlorella pyrenoidosa for saturated and unsaturated fatty acids production
  14. Klang - Medium - Natur
  15. The programme on ecosystem change and society (PECS) – a decade of deepening social-ecological research through a place-based focus
  16. Laufbahntheorien: Oft führt der Zufall zum Erfolg
  17. Nucleation kinetics of the γ-phase in a binary Mg-Al alloy
  18. A comprehensive assessment of ecosystem services
  19. Understanding the Greenhouse Effect by Embodiment
  20. OH radical reactivity of airborne terbuthylazine adsorbed on inert aerosol
  21. Fabeln der Finalfiguren
  22. Marahrens, August (1875-1950)
  23. Scaffolding inquiry-based science and chemistry education in inclusive classrooms
  24. EMA-Links
  25. Convention on the Prevention and Punishment of the Crime of Genocide
  26. Non-native Douglas fir (Pseudotsuga menziesii) promotes sentinel prey attack rates in Central European forests
  27. In silico toxicology protocols
  28. Reallabor versus Realexperiment
  29. DialogueMaps
  30. Mythos als Aufklärung
  31. Kooperativ forschen und Rechtschreibunterricht entwickeln
  32. Constructing the European Space Policy
  33. Democracy and the global spread of progressive taxes
  34. Pragmatic development and stay abroad
  35. The Place of the Symbolic
  36. Von Boccaccio zu Hartmut Lange
  37. Druckmedien
  38. A typology of actors and their strategies in multi-scale governance of wind turbine conflict within forests
  39. Uncertainty and sustainability in the management of rangelands
  40. Islamophobia
  41. Information system on literature in the field of ICT for environmental sustainability