Capitalizing on natural language processing (NLP) to automate the evaluation of coach implementation fidelity in guided digital cognitive-behavioral therapy (GdCBT)

Research output: Journal contributionsJournal articlesResearchpeer-review

Authors

  • Nur Hani Zainal
  • Regina Eckhardt
  • Gavin N. Rackoff
  • Ellen E. Fitzsimmons-Craft
  • Elsa Rojas-Ashe
  • Craig Barr Taylor
  • Burkhardt Funk
  • Daniel Eisenberg
  • Denise E. Wilfley
  • Michelle G. Newman

Background As the use of guided digitally-delivered cognitive-behavioral therapy (GdCBT) grows, pragmatic analytic tools are needed to evaluate coaches' implementation fidelity. Aims We evaluated how natural language processing (NLP) and machine learning (ML) methods might automate the monitoring of coaches' implementation fidelity to GdCBT delivered as part of a randomized controlled trial. Method Coaches served as guides to 6-month GdCBT with 3,381 assigned users with or at risk for anxiety, depression, or eating disorders. CBT-trained and supervised human coders used a rubric to rate the implementation fidelity of 13,529 coach-to-user messages. NLP methods abstracted data from text-based coach-to-user messages, and 11 ML models predicting coach implementation fidelity were evaluated. Results Inter-rater agreement by human coders was excellent (intra-class correlation coefficient 980-.992). Coaches achieved behavioral targets at the start of the GdCBT and maintained strong fidelity throughout most subsequent messages. Coaches also avoided prohibited actions (e.g. reinforcing users' avoidance). Sentiment analyses generally indicated a higher frequency of coach-delivered positive than negative sentiment words and predicted coach implementation fidelity with acceptable performance metrics (e.g. area under the receiver operating characteristic curve [AUC] = 74.48%). The final best-performing ML algorithms that included a more comprehensive set of NLP features performed well (e.g. AUC = 76.06%). Conclusions NLP and ML tools could help clinical supervisors automate monitoring of coaches' implementation fidelity to GdCBT. These tools could maximize allocation of scarce resources by reducing the personnel time needed to measure fidelity, potentially freeing up more time for high-quality clinical care.

Original languageEnglish
Article numbere106
JournalPsychological Medicine
Volume55
ISSN0033-2917
DOIs
Publication statusPublished - 02.04.2025

Bibliographical note

Publisher Copyright:
© The Author(s), 2025. Published by Cambridge University Press.

    Research areas

  • anxiety, depression, digital mental health intervention, eating disorders, guided internet-delivered cognitive-behavioral therapy, implementation fidelity, machine learning, natural language processing
  • Business informatics

Recently viewed

Publications

  1. Home range size and resource use of breeding and non-breeding white storks along a land use gradient
  2. Dynamic efficiency and path dependencies in venture capital markets
  3. University-linked programmes for sustainable entrepreneurship and regional development
  4. Thanking and responding to thanks in American English: Language patterning and contextual appropriateness
  5. Predicting the future performance of soccer players
  6. The Crowd in Flux
  7. Passion, Performance and Soberness
  8. Transformative Impulse
  9. Microstructure-Oriented Fatigue Crack Propagation in Two Cast Mg–Al–Ba–Ca Alloys
  10. Modernization
  11. »CO2 causes a hole in the atmosphere« Using laypeople’s conceptions as a starting point to communicate climate change
  12. The hidden hand that shapes conceptual understanding: Choosing effective representations for teaching cell division and climate change
  13. On the geometric control of internal forces in power grasps
  14. Moving beyond unlearning unsustainable consumption
  15. Commitment Strategies for Sustainability
  16. Utilization of phenolic compounds by microalgae
  17. Transition management as an approach to deal with climate change
  18. Review of transit data sources
  19. Multiple forest structural elements are needed to promote beetle biomass, diversity and abundance
  20. Leaf Nutritional Content, Tree Richness, and Season Shape the Caterpillar Functional Trait Composition Hosted by Trees
  21. The Automated will
  22. Dynamic capabilities?
  23. Processability of Mg-Gd Powder via Friction Extrusion
  24. Trade-off between Production and Inventory Costs with Respect to a Given Demand Situation
  25. Understanding the bright side and the dark side of telework