1. //
  2. Start
  3. //
  4. Publications
  5. Metrics for Experimentation Programs: Ca...
  6. Output formats
  7. //

Metrics for Experimentation Programs: Categories, Benefits and Challenges

Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review

Standard

Metrics for Experimentation Programs: Categories, Benefits and Challenges. / Stotz, Nils; Drews, Paul.
Agile Processes in Software Engineering and Extreme Programming: 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings. ed. / Sibylle Peter; Martin Kropp; Ademar Aguiar; Craig Anslow; Maria Ilaria Lunesu; Andrea Pinna. Cham: Springer Nature, 2025. p. 210-225 ( Lecture Notes in Business Information Processing; Vol. 545).

Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review

Harvard

Stotz, N & Drews, P 2025, Metrics for Experimentation Programs: Categories, Benefits and Challenges. in S Peter, M Kropp, A Aguiar, C Anslow, MI Lunesu & A Pinna (eds), Agile Processes in Software Engineering and Extreme Programming: 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings. Lecture Notes in Business Information Processing, vol. 545, Springer Nature, Cham, pp. 210-225, 26th International Conference on Agile Software Development - XP 2025, Brugg-Windisch, Switzerland, 02.06.25. https://doi.org/10.1007/978-3-031-94544-1_15

APA

Stotz, N., & Drews, P. (2025). Metrics for Experimentation Programs: Categories, Benefits and Challenges. In S. Peter, M. Kropp, A. Aguiar, C. Anslow, M. I. Lunesu, & A. Pinna (Eds.), Agile Processes in Software Engineering and Extreme Programming: 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings (pp. 210-225). ( Lecture Notes in Business Information Processing; Vol. 545). Springer Nature. https://doi.org/10.1007/978-3-031-94544-1_15

Vancouver

Stotz N, Drews P. Metrics for Experimentation Programs: Categories, Benefits and Challenges. In Peter S, Kropp M, Aguiar A, Anslow C, Lunesu MI, Pinna A, editors, Agile Processes in Software Engineering and Extreme Programming: 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings. Cham: Springer Nature. 2025. p. 210-225. ( Lecture Notes in Business Information Processing). Epub 2025 May 29. doi: 10.1007/978-3-031-94544-1_15

Bibtex

@inbook{63d2eb4c93a447a4a2b9e81b2a41c7a6,
title = "Metrics for Experimentation Programs: Categories, Benefits and Challenges",
abstract = "Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.",
keywords = "A/B Testing, Continuous Experimentation, Experimentation Metrics, Program-Level Evaluation, Business informatics",
author = "Nils Stotz and Paul Drews",
note = "Publisher Copyright: {\textcopyright} The Author(s) 2025.; 26th International Conference on Agile Software Development - XP 2025, XP 2025 ; Conference date: 02-06-2025 Through 05-06-2025",
year = "2025",
doi = "10.1007/978-3-031-94544-1_15",
language = "English",
isbn = "978-3-031-94543-4",
series = " Lecture Notes in Business Information Processing",
publisher = "Springer Nature",
pages = "210--225",
editor = "Sibylle Peter and Martin Kropp and Ademar Aguiar and Craig Anslow and Lunesu, {Maria Ilaria} and Andrea Pinna",
booktitle = "Agile Processes in Software Engineering and Extreme Programming",
address = "Switzerland",

}

RIS

TY - CHAP

T1 - Metrics for Experimentation Programs: Categories, Benefits and Challenges

AU - Stotz, Nils

AU - Drews, Paul

N1 - Publisher Copyright: © The Author(s) 2025.

PY - 2025

Y1 - 2025

N2 - Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.

AB - Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.

KW - A/B Testing

KW - Continuous Experimentation

KW - Experimentation Metrics

KW - Program-Level Evaluation

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=105007523247&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-94544-1_15

DO - 10.1007/978-3-031-94544-1_15

M3 - Article in conference proceedings

SN - 978-3-031-94543-4

T3 - Lecture Notes in Business Information Processing

SP - 210

EP - 225

BT - Agile Processes in Software Engineering and Extreme Programming

A2 - Peter, Sibylle

A2 - Kropp, Martin

A2 - Aguiar, Ademar

A2 - Anslow, Craig

A2 - Lunesu, Maria Ilaria

A2 - Pinna, Andrea

PB - Springer Nature

CY - Cham

T2 - 26th International Conference on Agile Software Development - XP 2025

Y2 - 2 June 2025 through 5 June 2025

ER -

Other publications by the same author(s)

AI-Enhanced Literature Reviews: Connecting Emerging Phenomena and Bodies of Knowledge

Naqvi, S. A. A., Zimmer, M. P., Kauschinger, M., Drews, P. & Basole, R. C., 2026, (Accepted/In press) Proceedings of HICSS 2026.

Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review

Aligning Experimentation with Product Operations: A Taxonomy for Structuring Experimentation Teams

Stotz, N., Labay, B., Vermeer, L. & Drews, P., 2026, Software Engineering and Advanced Applications - 51st Euromicro Conference, SEAA 2025, Proceedings: 51st Euromicro Conference, SEAA 2025 Salerno, Italy, September 10–12, 2025 Proceedings, Part III. Taibi, D. & Smite, D. (eds.). Cham: Springer Nature Switzerland AG, Vol. 3. p. 23-38 16 p. (Lecture Notes in Computer Science; vol. 16083).

Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review

Conveying the Ethics of Artificial Intelligence in K–12 and Academia: A Systematic Review of Teaching Methods

Tschoppe, N. J., Katsarov, J., Drews, P. & Trittin-Ulbrich, H., 07.01.2025, Proceedings of the 58th Hawaii International Conference on System Sciences: Hilton Waikoloa Village, January 7-10, 2025. Bui, T. X. (ed.). Honolulu: University of Hawaii at Manoa, p. 4744-4753 10 p. (Hawaii International Conference on System Sciences (HICSS); vol. 2025).

Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review

Digital leadership routines: Understanding the role of artifacts in digital leadership development

Eberl, J. K., Zimmer, M. P. & Drews, P., 12.2025, In: Information and Organization. 35, 4, 100599.

Research output: Journal contributions › Journal articles › Research › peer-review

Govern Like You Mean It: Embedding Data Ethics Practices within Data Governance

Richter, K., Zimmer, M. P. & Drews, P., 2025, AMCIS 2025 Proceedings. Atlanta: AIS eLibrary, 10 p. 1265. (The AMCIS proceedings series; vol. 2025).

Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review

DOI

Recently viewed

Activities

  1. Merely Chocolate-Covered Broccoli? From Early CALL to Digital Game-Based Language Learning
  2. Spec­tral Ki­ne­tic Si­mu­la­ti­on of Ideal Mul­ti­po­le Re­so­nan­ce Probe
  3. Workshop on Negotiation and Conflict Resolution at University of Amsterdam - UVA 2018
  4. Surviving in the wild: determinants and practices of strategic initiative development
  5. The Use of Media in Intercultural Dialogue "dialogo_dialog"!: Investigation of a Research Event in Terms of Communication without Language
  6. Universität Wien
  7. Methodenworkshop KuBIn
  8. Coauthoring direction when voices are many and authority is ambiguous
  9. The Documentary Method - an Introduction
  10. Negotiators' Value Creation Between Parties, Across Time
  11. Changing learning environments at university? Comparing the learning strategies of non-traditional European students engaged in lifelong learning.
  12. Beyond Unity
  13. Conference "FICTION AND SIMULATION: Theories, Methods, Practices" - 2015
  14. Dealing with Climate Change. Calculus & Catastrophe in the Age of Simulation - 2015
  15. On the perception and effectiveness of the feedback quality from a digital learning platform
  16. Panel Cointegration Testing with Time Trend and Analysis of Money Demand in OECD Countries

Publications

  1. Dynamic capabilities and routinization
  2. Performance concepts and performance theory
  3. The Network Dynamics of Movements
  4. Using Daily Stretching to Counteract Performance Decreases as a Result of Reduced Physical Activity—A Controlled Trial
  5. Integrating resilience thinking and optimisation for conservation
  6. Reciprocal Relationships Between Dispositional Optimism and Work Experiences
  7. Machine Learning Applications in Convective Turbulence
  8. The link between in- and external rotation of the auditor and the quality of financial accounting and external audit
  9. Improving Flood Forecasting in a Developing Country
  10. Comparison of three methods of length compensation in a parallel kinematic and their equivalence conditions
  11. Anomalous Results in G-Factor Models
  12. A Lyapunov Approach to Set the Parameters of a PI-Controller to Minimise Velocity Oscillations in a Permanent Magnet Synchronous Motor Using Chopper Control for Electrical Vehicles
  13. Absolutely continuous random power series in reciprocals of Pisot numbers
  14. The role of spatial ability when fostering mental animation in multimedia learning
  15. Modeling the effect of workpiece temperature on micromagnetic high-speed-3MA-testing in case of AISI 4140
  16. Introduction
  17. Integrating inductive and deductive analysis to identify and characterize archetypical social-ecological systems and their changes
  18. Europe and the media: Changing structures in a changing context
  19. Effectiveness of the world network of biosphere reserves in maintaining forest ecosystem functions
  20. Analyzing Talk and Text II: Thematic Analysis
  21. Assessment of cognitive load in multimedia learning using dual-task methodology
  22. Cobalt in end-of-life products in the EU, where does it end up? - The MaTrace approach
  23. Semi-Supervised Generative Models for Multi-Agent Trajectories