TY - JOUR

T1 - Toward a modular evaluation approach of real-world laboratories

T2 - Findings from a literature review

AU - Kampfmann, Teresa

AU - Bernert, Philip

AU - Lang, Daniel J.

N1 - This work was supported by the German Federal Ministry of Education and Research [13ZS0058B]. Publisher Copyright: © 2022 The Author(s). Published by Oxford University Press.

PY - 2023/1/1

Y1 - 2023/1/1

N2 - The number of real-world laboratories (RwLs) as research settings to address sustainability problems by using collaborative and experimental approaches has been growing steadily over the past years. RwLs are widely considered promising settings for the production of action-orientated knowledge in order to contribute solving wicked real-world problems such as climate change. However, empirically tested evaluative approaches are rare. We argue that evaluations within RwLs are crucial for ensuring high-quality and impactful research, amplifying tested solution approaches, and driving innovative forms of transdisciplinary collaboration. At the same time, evaluations within RwLs seem difficult due to the multilayered structure of RwLs. In order to provide guidance regarding the evaluation in RwLs-for researchers and practitioners, we conduct a scoping literature review that encompasses a comprehensive understanding of RwLs. We identify studies in which the specific components of RwLs lab context, experiments, and interventions -are empirically evaluated. Drawing on our findings, we derive different types of approaches that can be used for evaluating RwLs in practice. Based on how labs, experiments, and interventions in real-world settings have been evaluated in practice so far, we suggest to design evaluation approaches depending on (1) what should exactly be evaluated in the specific RwL, (2) for which purpose, and (3) by whom. We hope this targeted, modular evaluation approach based on evaluation types found in literature will help actors engaged in RwLs to make evaluations more feasible and impactful.

AB - The number of real-world laboratories (RwLs) as research settings to address sustainability problems by using collaborative and experimental approaches has been growing steadily over the past years. RwLs are widely considered promising settings for the production of action-orientated knowledge in order to contribute solving wicked real-world problems such as climate change. However, empirically tested evaluative approaches are rare. We argue that evaluations within RwLs are crucial for ensuring high-quality and impactful research, amplifying tested solution approaches, and driving innovative forms of transdisciplinary collaboration. At the same time, evaluations within RwLs seem difficult due to the multilayered structure of RwLs. In order to provide guidance regarding the evaluation in RwLs-for researchers and practitioners, we conduct a scoping literature review that encompasses a comprehensive understanding of RwLs. We identify studies in which the specific components of RwLs lab context, experiments, and interventions -are empirically evaluated. Drawing on our findings, we derive different types of approaches that can be used for evaluating RwLs in practice. Based on how labs, experiments, and interventions in real-world settings have been evaluated in practice so far, we suggest to design evaluation approaches depending on (1) what should exactly be evaluated in the specific RwL, (2) for which purpose, and (3) by whom. We hope this targeted, modular evaluation approach based on evaluation types found in literature will help actors engaged in RwLs to make evaluations more feasible and impactful.

KW - evaluation

KW - real-world laboratories

KW - RwLs

KW - transdisciplinary sustainability research

KW - transformation research

KW - Transdisciplinary studies

KW - Sustainability sciences, Communication

UR - http://www.scopus.com/inward/record.url?scp=85149283111&partnerID=8YFLogxK

UR - https://www.mendeley.com/catalogue/7cb47cf4-17ab-356e-bb01-b63f2d874721/

U2 - 10.1093/reseval/rvac029

DO - 10.1093/reseval/rvac029

M3 - Scientific review articles

AN - SCOPUS:85149283111

VL - 32

SP - 128

EP - 143

JO - Research Evaluation

JF - Research Evaluation

SN - 0958-2029

IS - 1

ER -