Metrics for Experimentation Programs: Categories, Benefits and Challenges

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.
Original languageEnglish
Title of host publicationAgile Processes in Software Engineering and Extreme Programming : 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings
EditorsSibylle Peter, Martin Kropp, Ademar Aguiar, Craig Anslow, Maria Ilaria Lunesu, Andrea Pinna
Number of pages16
Place of PublicationCham
PublisherSpringer Nature
Publication date2025
Pages210-225
ISBN (print)978-3-031-94543-4
ISBN (electronic)978-3-031-94544-1
DOIs
Publication statusPublished - 2025
Event26th International Conference on Agile Software Development - XP 2025 - Brugg-Windisch, Switzerland
Duration: 02.06.202505.06.2025

Bibliographical note

Publisher Copyright:
© The Author(s) 2025.

Recently viewed

Activities

  1. Learning through play? Evaluating digital games for language learning
  2. Peter G. Mahaffy
  3. Conference on Participatory Approaches in Science & Technology - PATH 2006
  4. International Workshop - Pragmatic Markers, Discourse Markers and Modal Particles: What do we know and where do we go from here?
  5. Conference presentation: The Relationship between the Internal Audit Function and the Audit Committee
  6. Denoising and Harmonic Detection Using Libraries of Nonorthogonal Trigonometric Bases
  7. Mutual Learning and Knowledge Integration in Transdisciplinary Development Teams: Empirical Findings about a Collaborative Format in Teacher Education
  8. Performing Contracts: BDSM, Performance Art and Aesthetics of Complexity
  9. Towards an International Keywords for Children's Literature (Roundtable): Presenters: Lissa Paul, Philip Nel, Nina Alonso, Nina Christensen, Francesca Orestano and Emer O’Sullivan
  10. Coupled component modelling for inter- and transdisciplinariy climate change impact reserach: dimensions of integration and examples of interface design
  11. Workshop „Meta-Image Day 2012”
  12. Workshop on Family Migration Processes in a Comparative Perspective - 2018
  13. From Podcast to Simulation Projects - Web 2.0 Projects in the Secondary EFL Classroom
  14. Shifting Backstages and Frontlines of Embodiment
  15. The efficacy of an internet- and mobile-based stress management intervention in employees in different guidance formats: the results of three randomised controlled trials
  16. Empirical Insights into Working in Research-Practice Partnerships: New Findings on Motivation, Co-Constructive Collaboration and Learning Effects
  17. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (Zeitschrift)

Publications

  1. Scholarly Question Answering Using Large Language Models in the NFDI4DataScience Gateway
  2. Assessment of university students’ understanding of abstract binary operations
  3. Faulty Process Detection Using Machine Learning Techniques
  4. Formative Perspectives on the Relation Between CSR Communication and CSR Practices
  5. Active learning for network intrusion detection
  6. Complexity of traffic scenes and EEG-measures of processing workload in car driving
  7. Constraints are the solution, not the problem
  8. Effect of gap distortion on the field splitting of collective modes in superfluid He3-B
  9. Jackson networks in nonautonomous random environments
  10. The generative drawing principle in multimedia learning
  11. Aspect-oriented software development
  12. Using corpus-linguistic methods to track longitudinal development
  13. The buffering effect of selection, optimization, and compensation strategy use on the relationship between problem solving demands and occupational well-being
  14. Performance of process-based models for simulation of grain N in crop rotations across Europe
  15. Self-tuning of a kalman filter applied in a DC drive and in a kalman-based sensor
  16. Effectiveness of a Web-Based Cognitive Behavioural Intervention for Subthreshold Depression
  17. Overcoming Multi-legacy Application Challenges through Building Dynamic Capabilities for Low-Code Adoption
  18. Constraint breeds creativity
  19. Gaining deep leverage? Reflecting and shaping real-world lab impacts through leverage points
  20. Action rate models for predicting actions in soccer
  21. Correlation of Microstructure and Local Mechanical Properties Along Build Direction for Multi-layer Friction Surfacing of Aluminum Alloys
  22. Learning from Erroneous Examples: When and How do Students Benefit from them?
  23. Perception and Inference
  24. Action Errors, Error Management, and Learning in Organizations
  25. From Knowledge to Application
  26. Development and application of a laboratory flux measurement system (LFMS) for the investigation of the kinetics of mercury emissions from soils
  27. How, when and why do negotiators use reference points?