Metrics for Experimentation Programs: Categories, Benefits and Challenges
Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review
Standard
Agile Processes in Software Engineering and Extreme Programming: 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings. ed. / Sibylle Peter; Martin Kropp; Ademar Aguiar; Craig Anslow; Maria Ilaria Lunesu; Andrea Pinna. Cham: Springer Nature, 2025. p. 210-225 ( Lecture Notes in Business Information Processing; Vol. 545).
Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review
Harvard
APA
Vancouver
Bibtex
}
RIS
TY - CHAP
T1 - Metrics for Experimentation Programs: Categories, Benefits and Challenges
AU - Stotz, Nils
AU - Drews, Paul
N1 - Publisher Copyright: © The Author(s) 2025.
PY - 2025
Y1 - 2025
N2 - Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.
AB - Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.
KW - A/B Testing
KW - Continuous Experimentation
KW - Experimentation Metrics
KW - Program-Level Evaluation
KW - Business informatics
UR - http://www.scopus.com/inward/record.url?scp=105007523247&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-94544-1_15
DO - 10.1007/978-3-031-94544-1_15
M3 - Article in conference proceedings
SN - 978-3-031-94543-4
T3 - Lecture Notes in Business Information Processing
SP - 210
EP - 225
BT - Agile Processes in Software Engineering and Extreme Programming
A2 - Peter, Sibylle
A2 - Kropp, Martin
A2 - Aguiar, Ademar
A2 - Anslow, Craig
A2 - Lunesu, Maria Ilaria
A2 - Pinna, Andrea
PB - Springer Nature
CY - Cham
T2 - 26th International Conference on Agile Software Development - XP 2025
Y2 - 2 June 2025 through 5 June 2025
ER -