Metrics for Experimentation Programs: Categories, Benefits and Challenges

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.
Original languageEnglish
Title of host publicationAgile Processes in Software Engineering and Extreme Programming : 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings
EditorsSibylle Peter, Martin Kropp, Ademar Aguiar, Craig Anslow, Maria Ilaria Lunesu, Andrea Pinna
Number of pages16
Place of PublicationCham
PublisherSpringer Nature
Publication date2025
Pages210-225
ISBN (print)978-3-031-94543-4
ISBN (electronic)978-3-031-94544-1
DOIs
Publication statusPublished - 2025
Event26th International Conference on Agile Software Development - XP 2025 - Brugg-Windisch, Switzerland
Duration: 02.06.202505.06.2025

Bibliographical note

Publisher Copyright:
© The Author(s) 2025.

Recently viewed

Activities

  1. Coauthoring an interorganizational collaboration: Exploring multi-voicedness and introducing spatiotemporal orientations
  2. Plasma shock wave simulation for laser shock processing
  3. Transformations 2017
  4. Learning through play? Evaluating digital games for language learning
  5. THE EUROPEAN CONFERENCE ON MACHINE LEARNING & PRINCIPLES AND PRACTICE OF KNOWLEDGE DISCOVERY IN DATABASES 2017
  6. The Linguistic Complexity of Test Items: Differential Effects for Students With Low and High Language Proficiency
  7. Peter G. Mahaffy
  8. Ars Electronica
  9. Workshop Medzin I
  10. Performance resource depletion influence on performance: Advancing concepts and findings
  11. Assessing cognitive abilities for L2 learning: A review of accessible tools
  12. SPARQL Querying Benchmarks
  13. Conference on Participatory Approaches in Science & Technology - PATH 2006
  14. International Workshop - Pragmatic Markers, Discourse Markers and Modal Particles: What do we know and where do we go from here?
  15. Why do extreme work hours persist? Temporal uncoupling as a new way of seeing
  16. Simulation and Evaluation of Control Mechanisms for Mobile Robot Fulfillment Systems
  17. Presentation of the paper entitled "Design of a Real Time Path of Motion Control for Manufacturing Applications"
  18. Setting up camp: emerging organizing around contested issues in the context of covert and illegal living on campsites
  19. Denoising and Harmonic Detection Using Libraries of Nonorthogonal Trigonometric Bases
  20. Experiences on the theme of actions for sustainable development in the field of educational systems

Publications

  1. Robust Control of Excavation Mobile Robot with Dynamic Triangulation Vision
  2. Scholarly Question Answering Using Large Language Models in the NFDI4DataScience Gateway
  3. Designing and evaluating blended learning bridging courses in mathematics
  4. Modeling of lateness distributions depending on the sequencing method with respect to productivity effects
  5. What Makes for a Good Theory? How to Evaluate a Theory Using the Strength Model of Self-Control as an Example
  6. Evaluation of standard ERP software implementation approaches in terms of their capability for business process optimization
  7. Are all errors created equal?
  8. Do connectives improve the level of understandability in mathematical reality-based tasks?
  9. Executive function and Language Learning
  10. TARGET SETTING FOR OPERATIONAL PERFORMANCE IMPROVEMENTS - STUDY CASE -
  11. "And I Think That Is a Very Straightforward Way of Dealing With It''
  12. Design of an Energy Efficient Sensor Node for Wearable Applications
  13. Proxies
  14. An Adaptive and Optimized Switching Observer for Sensorless Control of an Electromagnetic Valve Actuator in Camless Internal Combustion Engines
  15. Collaborative open science as a way to reproducibility and new insights in primate cognition research
  16. Measuring cognitive load with subjective rating scales during problem solving
  17. Sensitivity to complexity - an important prerequisite of problem solving mathematics teaching
  18. The temporal pattern of creativity and implementation in teams
  19. Conceptions of problem solving mathematics teaching
  20. A reference architecture for the integration of EMIS and ERP-Systems
  21. Parametric finite element model and mechanical characterisation of electrospun materials for biomedical applications
  22. What´s in a net? or: The end of the average
  23. An integrative research framework for enabling transformative adaptation
  24. Faulty Process Detection Using Machine Learning Techniques