Clinical prediction models for psychotherapy outcomes have the potential to assist therapists in planning a patient’s individual treatment, but their clinical utility depends on maintaining predictive performance across settings. We assessed prediction of post-treatment remission in 2,002 patients with internalizing disorders treated across eight German university outpatient clinics. Penalized logistic-regression models were evaluated within clinics, in pooled clinic data, and using leave-one-clinic-out validation.
Within clinics, mean AUC was 0.67, balanced accuracy 0.62, and Brier score 0.21. In leave-one-clinic-out validation, mean AUC and balanced accuracy declined from 0.60 and 0.57 in corresponding held-out test samples to 0.53 and 0.52 in held-out clinics, respectively. The mean Brier score increased from 0.23 to 0.25, but this difference was not statistically significant (p = 0.18).
Cross-clinic performance loss was therefore clearest for discrimination and threshold-dependent classification, while overall probabilistic prediction error showed a smaller change. These findings highlight the broader challenge of transporting psychotherapy outcome predictions across heterogeneous clinical settings. Precision mental healthcare has the potential to enhance patient treatment efficacy.
Both patient diagnostic and prognostic stages can be augmented with the use of model derived predictions1,2. In a personalized approach to treatment selection, medical decisions are tailored based on the outcomes of similar patient profiles that have been observed in large data sets. This evidence-based approach allows the fit of a particular course of treatment to be calculated before starting therapy.
Treatment response and treatment remission prediction models in particular hold promise for optimizing treatment selection and a number of studies have shown that reasonable model predictive performance can be found for various mental disorders3, such as mood disorders4,5,6,7, anxiety disorders8,9,10, obsessive compulsive disorder11, and in a heterogeneous sample12,13. The value of predictive models in informing practitioner decisions is constrained by the quality of the resulting predictions. In addition, the promise of predictive models in optimizing patient treatment has been questioned due to methodological issues arising during model construction14.
Furthermore, small sample sizes may inflate model performance, with a number of studies showing that initially well performing models could not be replicated with larger sample sizes15. Another issue that has been called into question is the extent to which models that show good performance in one clinical setting generalize to new patients in other contexts16,17. Generalizability describes the extent to which a research finding can be applied to other settings other than the one that was used to test the initial finding.
Extract — continue reading at the source.
