Capitalizing on natural language processing (NLP) to automate the evaluation of coach implementation fidelity in guided digital cognitive-behavioral therapy (GdCBT)

Publikation: Beiträge in ZeitschriftenZeitschriftenaufsätzeForschungbegutachtet

Standard

Capitalizing on natural language processing (NLP) to automate the evaluation of coach implementation fidelity in guided digital cognitive-behavioral therapy (GdCBT). / Zainal, Nur Hani; Eckhardt, Regina; Rackoff, Gavin N. et al.
in: Psychological Medicine, Jahrgang 55, e106, 02.04.2025.

Publikation: Beiträge in ZeitschriftenZeitschriftenaufsätzeForschungbegutachtet

Harvard

APA

Zainal, N. H., Eckhardt, R., Rackoff, G. N., Fitzsimmons-Craft, E. E., Rojas-Ashe, E., Barr Taylor, C., Funk, B., Eisenberg, D., Wilfley, D. E., & Newman, M. G. (2025). Capitalizing on natural language processing (NLP) to automate the evaluation of coach implementation fidelity in guided digital cognitive-behavioral therapy (GdCBT). Psychological Medicine, 55, Artikel e106. https://doi.org/10.1017/S0033291725000340

Vancouver

Zainal NH, Eckhardt R, Rackoff GN, Fitzsimmons-Craft EE, Rojas-Ashe E, Barr Taylor C et al. Capitalizing on natural language processing (NLP) to automate the evaluation of coach implementation fidelity in guided digital cognitive-behavioral therapy (GdCBT). Psychological Medicine. 2025 Apr 2;55:e106. doi: 10.1017/S0033291725000340

Bibtex

@article{3cce202d951741bf8ca36f669098de6a,
title = "Capitalizing on natural language processing (NLP) to automate the evaluation of coach implementation fidelity in guided digital cognitive-behavioral therapy (GdCBT)",
abstract = "Background As the use of guided digitally-delivered cognitive-behavioral therapy (GdCBT) grows, pragmatic analytic tools are needed to evaluate coaches' implementation fidelity. Aims We evaluated how natural language processing (NLP) and machine learning (ML) methods might automate the monitoring of coaches' implementation fidelity to GdCBT delivered as part of a randomized controlled trial. Method Coaches served as guides to 6-month GdCBT with 3,381 assigned users with or at risk for anxiety, depression, or eating disorders. CBT-trained and supervised human coders used a rubric to rate the implementation fidelity of 13,529 coach-to-user messages. NLP methods abstracted data from text-based coach-to-user messages, and 11 ML models predicting coach implementation fidelity were evaluated. Results Inter-rater agreement by human coders was excellent (intra-class correlation coefficient 980-.992). Coaches achieved behavioral targets at the start of the GdCBT and maintained strong fidelity throughout most subsequent messages. Coaches also avoided prohibited actions (e.g. reinforcing users' avoidance). Sentiment analyses generally indicated a higher frequency of coach-delivered positive than negative sentiment words and predicted coach implementation fidelity with acceptable performance metrics (e.g. area under the receiver operating characteristic curve [AUC] = 74.48%). The final best-performing ML algorithms that included a more comprehensive set of NLP features performed well (e.g. AUC = 76.06%). Conclusions NLP and ML tools could help clinical supervisors automate monitoring of coaches' implementation fidelity to GdCBT. These tools could maximize allocation of scarce resources by reducing the personnel time needed to measure fidelity, potentially freeing up more time for high-quality clinical care.",
keywords = "anxiety, depression, digital mental health intervention, eating disorders, guided internet-delivered cognitive-behavioral therapy, implementation fidelity, machine learning, natural language processing, Business informatics",
author = "Zainal, {Nur Hani} and Regina Eckhardt and Rackoff, {Gavin N.} and Fitzsimmons-Craft, {Ellen E.} and Elsa Rojas-Ashe and {Barr Taylor}, Craig and Burkhardt Funk and Daniel Eisenberg and Wilfley, {Denise E.} and Newman, {Michelle G.}",
note = "Publisher Copyright: {\textcopyright} The Author(s), 2025. Published by Cambridge University Press.",
year = "2025",
month = apr,
day = "2",
doi = "10.1017/S0033291725000340",
language = "English",
volume = "55",
journal = "Psychological Medicine",
issn = "0033-2917",
publisher = "Cambridge University Press",

}

RIS

TY - JOUR

T1 - Capitalizing on natural language processing (NLP) to automate the evaluation of coach implementation fidelity in guided digital cognitive-behavioral therapy (GdCBT)

AU - Zainal, Nur Hani

AU - Eckhardt, Regina

AU - Rackoff, Gavin N.

AU - Fitzsimmons-Craft, Ellen E.

AU - Rojas-Ashe, Elsa

AU - Barr Taylor, Craig

AU - Funk, Burkhardt

AU - Eisenberg, Daniel

AU - Wilfley, Denise E.

AU - Newman, Michelle G.

N1 - Publisher Copyright: © The Author(s), 2025. Published by Cambridge University Press.

PY - 2025/4/2

Y1 - 2025/4/2

N2 - Background As the use of guided digitally-delivered cognitive-behavioral therapy (GdCBT) grows, pragmatic analytic tools are needed to evaluate coaches' implementation fidelity. Aims We evaluated how natural language processing (NLP) and machine learning (ML) methods might automate the monitoring of coaches' implementation fidelity to GdCBT delivered as part of a randomized controlled trial. Method Coaches served as guides to 6-month GdCBT with 3,381 assigned users with or at risk for anxiety, depression, or eating disorders. CBT-trained and supervised human coders used a rubric to rate the implementation fidelity of 13,529 coach-to-user messages. NLP methods abstracted data from text-based coach-to-user messages, and 11 ML models predicting coach implementation fidelity were evaluated. Results Inter-rater agreement by human coders was excellent (intra-class correlation coefficient 980-.992). Coaches achieved behavioral targets at the start of the GdCBT and maintained strong fidelity throughout most subsequent messages. Coaches also avoided prohibited actions (e.g. reinforcing users' avoidance). Sentiment analyses generally indicated a higher frequency of coach-delivered positive than negative sentiment words and predicted coach implementation fidelity with acceptable performance metrics (e.g. area under the receiver operating characteristic curve [AUC] = 74.48%). The final best-performing ML algorithms that included a more comprehensive set of NLP features performed well (e.g. AUC = 76.06%). Conclusions NLP and ML tools could help clinical supervisors automate monitoring of coaches' implementation fidelity to GdCBT. These tools could maximize allocation of scarce resources by reducing the personnel time needed to measure fidelity, potentially freeing up more time for high-quality clinical care.

AB - Background As the use of guided digitally-delivered cognitive-behavioral therapy (GdCBT) grows, pragmatic analytic tools are needed to evaluate coaches' implementation fidelity. Aims We evaluated how natural language processing (NLP) and machine learning (ML) methods might automate the monitoring of coaches' implementation fidelity to GdCBT delivered as part of a randomized controlled trial. Method Coaches served as guides to 6-month GdCBT with 3,381 assigned users with or at risk for anxiety, depression, or eating disorders. CBT-trained and supervised human coders used a rubric to rate the implementation fidelity of 13,529 coach-to-user messages. NLP methods abstracted data from text-based coach-to-user messages, and 11 ML models predicting coach implementation fidelity were evaluated. Results Inter-rater agreement by human coders was excellent (intra-class correlation coefficient 980-.992). Coaches achieved behavioral targets at the start of the GdCBT and maintained strong fidelity throughout most subsequent messages. Coaches also avoided prohibited actions (e.g. reinforcing users' avoidance). Sentiment analyses generally indicated a higher frequency of coach-delivered positive than negative sentiment words and predicted coach implementation fidelity with acceptable performance metrics (e.g. area under the receiver operating characteristic curve [AUC] = 74.48%). The final best-performing ML algorithms that included a more comprehensive set of NLP features performed well (e.g. AUC = 76.06%). Conclusions NLP and ML tools could help clinical supervisors automate monitoring of coaches' implementation fidelity to GdCBT. These tools could maximize allocation of scarce resources by reducing the personnel time needed to measure fidelity, potentially freeing up more time for high-quality clinical care.

KW - anxiety

KW - depression

KW - digital mental health intervention

KW - eating disorders

KW - guided internet-delivered cognitive-behavioral therapy

KW - implementation fidelity

KW - machine learning

KW - natural language processing

KW - Business informatics

UR - http://www.scopus.com/inward/record.url?scp=105001856822&partnerID=8YFLogxK

U2 - 10.1017/S0033291725000340

DO - 10.1017/S0033291725000340

M3 - Journal articles

C2 - 40170669

AN - SCOPUS:105001856822

VL - 55

JO - Psychological Medicine

JF - Psychological Medicine

SN - 0033-2917

M1 - e106

ER -

DOI

Zuletzt angesehen

Forschende

  1. Nadine Lüpschen

Publikationen

  1. A Motion-Sensorless Control for Intake Valves in Combustion Engines
  2. Use of lignins from sugarcane bagasse for assembling microparticles loaded with Azadirachta indica extracts for use as neem-based organic insecticides
  3. Skill learning as a concept in life-span developmental psychology
  4. How to Explain Major Policy Change Towards Sustainability? Bringing Together the Multiple Streams Framework and the Multilevel Perspective on Socio-Technical Transitions to Explore the German “Energiewende”
  5. Improving Human-Machine Interaction
  6. Adapting and evolving-learning place cooperation in change
  7. Continuous Casting with Mid-Process Alloying
  8. Theorizing path dependence
  9. Synthesis, self-assembly, bacterial and fungal toxicity, and preliminary biodegradation studies of a series of L-phenylalanine-derived surface-active ionic liquids
  10. Einführung in Grundlagen der theoretischen Informatik
  11. Learning processes for interpersonal competence development in project-based sustainability courses – insights from a comparative international study
  12. CASE via MS
  13. Heterogeneity and Diversity
  14. Inventory of biodegradation data of ionic liquids
  15. Gasteditorial
  16. Advancing science on the multiple connections between biodiversity, ecosystems and people
  17. Social group membership does not modulate automatic imitation in a contrastive multi-agent paradigm
  18. Anticipated imitation of multiple agents
  19. Three-dimensional microstructural analysis of Mg-Al-Zn alloys by synchrotron-radiation-based microtomography
  20. Entwicklung und realisierung eines computer-basierten lernprogramms zur GMP-schulung/Programm-entwicklung und benutzer-akzeptanz
  21. Perceptual latency priming
  22. Impacts of urban real-world labs: Insights from a co-evaluation process informed by structuration theory in Wuppertal-Mirke
  23. Targeted metabolomics of pellicle and saliva in children with different caries activity
  24. Public service media, innovation policy and the ‘crowding out’ problem
  25. Flexible Manufacturing of Concave–Convex Parts by Incremental Sheet Forming with Active Medium
  26. On melting summits
  27. Myth/Mythology
  28. Young children spontaneously recreate core properties of language in a new modality
  29. Over here and over there
  30. German Version of the Relationship Problems Questionnaire
  31. Magnús eiríksson