Developing a Process for the Analysis of User Journeys and the Prediction of Dropout in Digital Health Interventions: Machine Learning Approach

Research output: Journal contributionsConference article in journalResearchpeer-review

Authors

Background: User dropout is a widespread concern in the delivery and evaluation of digital (ie, web and mobile apps) health interventions. Researchers have yet to fully realize the potential of the large amount of data generated by these technology-based programs. Of particular interest is the ability to predict who will drop out of an intervention. This may be possible through the analysis of user journey data—self-reported as well as system-generated data—produced by the path (or journey) an individual takes to navigate through a digital health intervention.

Objective: The purpose of this study is to provide a step-by-step process for the analysis of user journey data and eventually to predict dropout in the context of digital health interventions. The process is applied to data from an internet-based intervention for insomnia as a way to illustrate its use. The completion of the program is contingent upon completing 7 sequential cores, which include an initial tutorial core. Dropout is defined as not completing the seventh core.

Methods: Steps of user journey analysis, including data transformation, feature engineering, and statistical model analysis and evaluation, are presented. Dropouts were predicted based on data from 151 participants from a fully automated web-based program (Sleep Healthy Using the Internet) that delivers cognitive behavioral therapy for insomnia. Logistic regression with L1 and L2 regularization, support vector machines, and boosted decision trees were used and evaluated based on their predictive performance. Relevant features from the data are reported that predict user dropout.

Results: Accuracy of predicting dropout (area under the curve [AUC] values) varied depending on the program core and the machine learning technique. After model evaluation, boosted decision trees achieved AUC values ranging between 0.6 and 0.9. Additional handcrafted features, including time to complete certain steps of the intervention, time to get out of bed, and days since the last interaction with the system, contributed to the prediction performance.

Conclusions: The results support the feasibility and potential of analyzing user journey data to predict dropout. Theory-driven handcrafted features increased the prediction performance. The ability to predict dropout at an individual level could be used to enhance decision making for researchers and clinicians as well as inform dynamic intervention regimens.
Original languageEnglish
Article numbere17738
JournalJournal of Medical Internet Research
Volume22
Issue number10
Number of pages20
ISSN1439-4456
DOIs
Publication statusPublished - 28.10.2020

Bibliographical note

Publisher Copyright:
©Vincent Bremer, Philip I Chow, Burkhardt Funk, Frances P Thorndike, Lee M Ritterband.

Documents

DOI

Recently viewed

Publications

  1. Trap nests for bees and wasps to analyse trophic interactions in changing environments—A systematic overview and user guide
  2. Digital Seriality as Structure and Process
  3. On Software, or the Persistence of Visual Knowledge.
  4. The impact of explicit references in computer supported collaborative learning: Evidence from eye movement analyses
  5. Missing links
  6. Hacking the Classroom
  7. How do students and teachers deal with mathematical modelling problems?
  8. Employing A-B tests for optimizing prices levels in e-commerce applications
  9. Influence of measurement errors on networks
  10. Determinants in the online distribution of digital content
  11. Does online-delivered Cognitive Behavioural Therapy for Insomnia improve insomnia severity in nurses working shifts? Protocol for a randomised-controlled trial
  12. Next level production networks
  13. Where do the data live?
  14. Unravelling insect declines: Can space replace time?
  15. Sustainable development and learning for sustainability through a regional network project
  16. Machine Learning and Data Mining for Sports Analytics
  17. Object-Oriented Construction Handbook
  18. A Semiparametric Approach for Modeling Not-Reached Items
  19. Microstructure, mechanical and functional properties of refill friction stir spot welds on multilayered aluminum foils for battery application
  20. Solvable problems or problematic solvability?
  21. Development and validation of the Later Life Work Index for successful management of an aging workforce
  22. Systematic distributions of interaction strengths across tree interaction networks yield positive diversity–productivity relationships
  23. Political discourse in the media
  24. Diversity and specificity of host-natural enemy interactions in an urban-rural interface
  25. Taming a Wicked Problem
  26. Optimum parameters and rate-controlling mechanisms for hot working of extruded Mg-3Sn-1Ca alloy
  27. Micro and Macro Perspectives in Organization Theory
  28. Response of saproxylic beetles to small-scale habitat connectivity depends on trophic levels
  29. How can employment relations in global value networks be managed towards social responsibility?
  30. Systemanalyse für Softwaresysteme
  31. MindMatters
  32. The use of the online Inverted Classroom Model for digital teaching with gamification in medical studies
  33. Smart cities, smart borders. Sensing networks and security in the urban space
  34. The Influence of Tree Diversity on Natural Enemies—a Review of the “Enemies” Hypothesis in Forests
  35. Der "getarnte" Arbeitnehmer-Geschäftsführer
  36. Anatomical and neuromuscular variables strongly predict maximum knee extension torque in healthy men
  37. Aligning the design of intermediary organisations with the ecosystem
  38. Children's interpretation of ambiguous pronouns based on prior discourse
  39. Klassensprachen - Some Preliminary Theses
  40. Organizational practices for the aging workforce
  41. Safer Spaces
  42. Microtomography on biomaterials using the harwi-2 beamline at desy
  43. Tormentil for active ulcerative colitis