Capitalizing on natural language processing (NLP) to automate the evaluation of coach implementation fidelity in guided digital cognitive-behavioral therapy (GdCBT)

Publikation: Beiträge in ZeitschriftenZeitschriftenaufsätzeForschungbegutachtet

Authors

  • Nur Hani Zainal
  • Regina Eckhardt
  • Gavin N. Rackoff
  • Ellen E. Fitzsimmons-Craft
  • Elsa Rojas-Ashe
  • Craig Barr Taylor
  • Burkhardt Funk
  • Daniel Eisenberg
  • Denise E. Wilfley
  • Michelle G. Newman

Background As the use of guided digitally-delivered cognitive-behavioral therapy (GdCBT) grows, pragmatic analytic tools are needed to evaluate coaches' implementation fidelity. Aims We evaluated how natural language processing (NLP) and machine learning (ML) methods might automate the monitoring of coaches' implementation fidelity to GdCBT delivered as part of a randomized controlled trial. Method Coaches served as guides to 6-month GdCBT with 3,381 assigned users with or at risk for anxiety, depression, or eating disorders. CBT-trained and supervised human coders used a rubric to rate the implementation fidelity of 13,529 coach-to-user messages. NLP methods abstracted data from text-based coach-to-user messages, and 11 ML models predicting coach implementation fidelity were evaluated. Results Inter-rater agreement by human coders was excellent (intra-class correlation coefficient 980-.992). Coaches achieved behavioral targets at the start of the GdCBT and maintained strong fidelity throughout most subsequent messages. Coaches also avoided prohibited actions (e.g. reinforcing users' avoidance). Sentiment analyses generally indicated a higher frequency of coach-delivered positive than negative sentiment words and predicted coach implementation fidelity with acceptable performance metrics (e.g. area under the receiver operating characteristic curve [AUC] = 74.48%). The final best-performing ML algorithms that included a more comprehensive set of NLP features performed well (e.g. AUC = 76.06%). Conclusions NLP and ML tools could help clinical supervisors automate monitoring of coaches' implementation fidelity to GdCBT. These tools could maximize allocation of scarce resources by reducing the personnel time needed to measure fidelity, potentially freeing up more time for high-quality clinical care.

OriginalspracheEnglisch
Aufsatznummere106
ZeitschriftPsychological Medicine
Jahrgang55
ISSN0033-2917
DOIs
PublikationsstatusErschienen - 02.04.2025

Bibliographische Notiz

Publisher Copyright:
© The Author(s), 2025. Published by Cambridge University Press.

DOI

Zuletzt angesehen

Forschende

  1. Mareike Förster

Publikationen

  1. Rapid Prototyping of a Mechatronic Engine Valve Controller for IC Engines
  2. Dietary patterns of children on three indigenous societies
  3. Integrating work place and social environment into the measurement of work ability
  4. Detection of oscillations with application in the pantograph control
  5. Solution for the direct kinematics problem of the general stewart-gough platform by using only linear actuators’ orientations
  6. „Ist das dein Handy oder vibrierst du?“
  7. Improving collaboration between ecosystem service communities and the IPBES science-policy platform
  8. The utility of macroecological rules for microbial biogeography
  9. Collaboration and Open Science Initiatives in Primate Research
  10. Non-target Analysis and Chemometric Evaluation of a Passive Sampler Monitoring of Small Streams
  11. The Multiple Self Objection to the Prudential Lifespan Account
  12. Random year intercepts in mixed models help to assess uncertainties in insect population trends
  13. Friction riveting of 3D printed polyamide 6 with AA 6056-T6
  14. Indicators for relational values of nature’s contributions to good quality of life
  15. Old Wine in New Bottles? The Case of Self-compassion and Neuroticism
  16. Reconfigurable Control System for Plants with Variable Structure
  17. Fixed-term contracts and employment adjustment
  18. Microstructure by design
  19. Klimasimulation
  20. Differences in adaptation to light and temperature extremes of Chlorella sorokiniana strains isolated from a wastewater lagoon
  21. The potential impacts of insecticides on the life-history traits of bees and the consequences for pollination
  22. Mapping the Imaginary - Maps in Fantasy Role-Playing-Games
  23. Velocity planning to optimize traction losses in a City-Bus Equipped with Permanent Magnet Three-Phase Synchronous Motors
  24. Technikvergessenheit?
  25. Histological Comparison of New Biodegradable Magnesium-Based Implants for Maxillofacial Applications
  26. Automated Delivery
  27. The role of tree crown on the performance of trees at individual and community levels
  28. Why protect nature? Rethinking values and the environment