Capitalizing on natural language processing (NLP) to automate the evaluation of coach implementation fidelity in guided digital cognitive-behavioral therapy (GdCBT)

Publikation: Beiträge in ZeitschriftenZeitschriftenaufsätzeForschungbegutachtet

Authors

  • Nur Hani Zainal
  • Regina Eckhardt
  • Gavin N. Rackoff
  • Ellen E. Fitzsimmons-Craft
  • Elsa Rojas-Ashe
  • Craig Barr Taylor
  • Burkhardt Funk
  • Daniel Eisenberg
  • Denise E. Wilfley
  • Michelle G. Newman

Background As the use of guided digitally-delivered cognitive-behavioral therapy (GdCBT) grows, pragmatic analytic tools are needed to evaluate coaches' implementation fidelity. Aims We evaluated how natural language processing (NLP) and machine learning (ML) methods might automate the monitoring of coaches' implementation fidelity to GdCBT delivered as part of a randomized controlled trial. Method Coaches served as guides to 6-month GdCBT with 3,381 assigned users with or at risk for anxiety, depression, or eating disorders. CBT-trained and supervised human coders used a rubric to rate the implementation fidelity of 13,529 coach-to-user messages. NLP methods abstracted data from text-based coach-to-user messages, and 11 ML models predicting coach implementation fidelity were evaluated. Results Inter-rater agreement by human coders was excellent (intra-class correlation coefficient 980-.992). Coaches achieved behavioral targets at the start of the GdCBT and maintained strong fidelity throughout most subsequent messages. Coaches also avoided prohibited actions (e.g. reinforcing users' avoidance). Sentiment analyses generally indicated a higher frequency of coach-delivered positive than negative sentiment words and predicted coach implementation fidelity with acceptable performance metrics (e.g. area under the receiver operating characteristic curve [AUC] = 74.48%). The final best-performing ML algorithms that included a more comprehensive set of NLP features performed well (e.g. AUC = 76.06%). Conclusions NLP and ML tools could help clinical supervisors automate monitoring of coaches' implementation fidelity to GdCBT. These tools could maximize allocation of scarce resources by reducing the personnel time needed to measure fidelity, potentially freeing up more time for high-quality clinical care.

OriginalspracheEnglisch
Aufsatznummere106
ZeitschriftPsychological Medicine
Jahrgang55
ISSN0033-2917
DOIs
PublikationsstatusErschienen - 02.04.2025

Bibliographische Notiz

Publisher Copyright:
© The Author(s), 2025. Published by Cambridge University Press.

DOI

Zuletzt angesehen

Forschende

  1. Lukas Stolz

Publikationen

  1. Rapid Prototyping of a Mechatronic Engine Valve Controller for IC Engines
  2. Constitutions, Democratic Self-Determination and the Institutional Empowerment of Future Generations: Mitigating an Aporia
  3. Ticio Escobar
  4. Resolving conflicts between people and over time in the transformation toward sustainability
  5. Lyapunov stability analysis to set up a saturating PI controller with anti-windup for a mass flow system
  6. U-model-based dynamic inversion control for quadrotor UAV systems
  7. Systematic risk behavior in cyclical industries
  8. Bridging scenario planning and backcasting
  9. On the Existence of Digital Objects
  10. Release of monomers from four different composite materials after halogen and LED curing
  11. System and action theory
  12. Audio-Hacks
  13. Internet: Impact and Potential for Learning and Instruction
  14. Modeling and Simulation of Electrochemical Cells under Applied Voltage
  15. Controlling a Bank Model Economy by Sliding Mode Control with Help of Kalman Filter
  16. Does ESG performance have an impact on financial performance?
  17. An empirical investigation of experiences and the link between a servicedominant logic mindset, competitive advantage, and performance of nonprofit organizations
  18. How and Why Different Forms of Expertise Moderate Anchor Precision in Price Decisions
  19. An empirical note on commuting distance and sleep during workweek and weekend
  20. Dietary patterns of children on three indigenous societies
  21. Predictive mapping of plant species and communities using GIS and Landsat data in a southern Mongolian mountain range
  22. Der FFB-Server mit Microsoft Windows Server 2003
  23. Recurring patterns and blueprints of industrial symbioses as structural units for an it tool
  24. Shifts in plant functional trait dynamics in relation to soil microbiome in modern and wild barley
  25. Putting Architecture in its Social Space: the Fields and Skills of Planning Maastricht
  26. Detection of oscillations with application in the pantograph control
  27. Revegetation in agricultural areas: the development of structural complexity and floristic diversity
  28. The Integration of Wheelchair Users in Team Handball