Metrics for Experimentation Programs: Categories, Benefits and Challenges

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.
Original languageEnglish
Title of host publicationAgile Processes in Software Engineering and Extreme Programming : 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings
EditorsSibylle Peter, Martin Kropp, Ademar Aguiar, Craig Anslow, Maria Ilaria Lunesu, Andrea Pinna
Number of pages16
Place of PublicationCham
PublisherSpringer Nature
Publication date2025
Pages210-225
ISBN (print)978-3-031-94543-4
ISBN (electronic)978-3-031-94544-1
DOIs
Publication statusPublished - 2025
Event26th International Conference on Agile Software Development - XP 2025 - Brugg-Windisch, Switzerland
Duration: 02.06.202505.06.2025

Bibliographical note

Publisher Copyright:
© The Author(s) 2025.

Recently viewed

Publications

  1. A Computational Research System for the History of Science
  2. How to support synchronous net-based learning discourses
  3. How, when and why do negotiators use reference points?
  4. Optimal dynamic scale and structure of a multi-pollution economy
  5. Masked autoencoder for multiagent trajectories
  6. Evaluation of standard ERP software implementation approaches in terms of their capability for business process optimization
  7. Vergütung, variable
  8. Design of an Energy Efficient Sensor Node for Wearable Applications
  9. Tuning kalman filter in linear systems
  10. Wozu in Tönen denken?
  11. Sensitivity to complexity - an important prerequisite of problem solving mathematics teaching
  12. E-stability and stability of adaptive learning in models with asymmetric information
  13. Applying the Three Horizons approach in local and regional scenarios to support policy coherence in SDG implementation
  14. Technological System and the Problem of Desymbolization
  15. Differences of Four Work-Related Behavior and Experience Patterns in Work Ability and Other Work-Related Perceptions in a Finance Company
  16. Tree diversity and mycorrhizal type co-determine multitrophic ecosystem functions
  17. Making the most out of timeseries symptom data
  18. Performance Saga: Interview 01
  19. Study of fuzzy controllers performance
  20. Towards a Service-Oriented Architecture for Production Planning and Control
  21. Machine Learning Applications
  22. An Outcome-Oriented, Social-Ecological Framework for Assessing Protected Area Effectiveness
  23. 8th challenge on question answering over linked data (QALD-8)
  24. Project and Design of a Catamaran Prototype with Aerial Propulsion System