Sudden gains in routine clinical care: application of a permutation test for trauma-focused cognitive behavioural therapy

ABSTRACT

Background: Sudden gains, defined as large and stable improvements of psychopathological symptoms, are a ubiquitous phenomenon in psychotherapy. They have been shown to occur across several clinical contexts and to be associated with better short-term and long-term treatment outcome. However, the approach of sudden gains has been criticized for its tautological character: sudden gains are included in the computation of treatment outcomes, ultimately resulting in a circular conclusion. Furthermore, some authors criticize sudden gains as merely being random fluctuations.

Objective: Use of efficient methods to evaluate whether the amount of sudden gains in a given sample lies above chance level.

Method: We used permutation tests in a sample of 85 patients with posttraumatic stress disorder (PTSD) treated with trauma-focused cognitive behaviour therapy in routine clinical care. Scores of self-reported PTSD symptom severity were permuted 10.000 times within sessions and between participants to receive a random distribution.

Results: Altogether, 18 participants showed a total of 24 sudden gains within the first 20 sessions. The permutation test yielded that the frequency of sudden gains was not beyond chance level. No significant predictors of sudden gains were identified and sudden gains in general were not predictive of treatment outcome. However, subjects with early sudden gains had a significantly lower symptom severity after treatment.

Conclusions: Our data suggest that a significant proportion of sudden gains are due to chance. Further research is needed on the differential effects of early and late sudden gains.

HIGHLIGHTS

• Treatment-related sudden gains exhibit clinical significance when their manifestation is above chance level.

• We used permutation tests to examine their occurrence in trauma-focused cognitive behaviour therapy as applied in a naturalistic treatment setting.

• The occurrence of sudden gains in general was not significantly higher than chance, yet early sudden gains were associated with improved treatment outcome.

Antecedentes: Los progresos súbitos, definidos como mejorías amplias y estables en los síntomas psicopatológicos, son un fenómeno ubicuo en las psicoterapias. Se ha demostrado que suceden en diferentes contextos clínicos y que están asociados a mejores respuestas al tratamiento al corto y al largo plazo. Sin embargo, el abordaje de los progresos súbitos ha sido criticado por su carácter tautológico: Los progresos súbitos están incluidos en el cálculo de los resultados de los tratamientos, lo que resulta en una conclusión circular. Además, algunos autores realizan la crítica que los progresos súbitos no son sino meras fluctuaciones debidas al azar.

Objetivo: Usar métodos eficientes para evaluar si la cantidad de progresión súbita en una muestra determinada está por encima del punto de corte atribuido al azar.

Método: Empleamos pruebas de permutación en una muestra de 85 pacientes con el diagnóstico del trastorno de estrés postraumático (TEPT) en tratamiento con terapia cognitiva conductual centrada en el trauma en la atención clínica de rutina. Los puntajes de severidad de las escalas clínicas autorreportadas para TEPT fueron permutadas 10.000 veces entre las sesiones y entre los participantes para recibir una distribución aleatoria.

Resultados: En conjunto, 18 participantes mostraron un total de 24 progresos súbitos dentro de las primeras 20 sesiones. Las pruebas de permutación mostraron que la frecuencia de los progresos súbitos no se encontraba más allá del punto de corte atribuido al azar. Se identificaron predictores no significativos de progreso súbito, y el progreso súbito en general no fue predictor del resultado del tratamiento. Sin embargo, los sujetos con progresos súbitos tempranos obtuvieron significativamente una menor severidad en los síntomas luego del tratamiento.

Conclusiones: La información sugiere que una proporción significativa de progresos súbitos son debidas al azar. Se necesita más investigación en la diferenciación de los efectos de los progresos súbitos tempranos y tardíos.

KEYWORDS:

• Sudden gains
• PTSD
• routine clinical care
• trauma-focused treatment
• psychotherapy

PALABRAS CLAVE:

• Progresos súbitos
• TEPT
• atención clínica de rutina, tratamiento enfocado en trauma
• psicoterapia

1. Introduction

A wealth of clinical research has focused on sudden gains (SGs) that have been defined as large and stable reductions of psychopathological symptoms (Shalom & Aderka, 2020). Initially, SGs were defined by Tang and DeRubeis (1999) as large reductions in symptoms, which fulfil the criteria of being: 1) large in absolute magnitude, 2) large in relation to the previous symptom score (reduction of at least 25%) and 3) stable in relation to symptom fluctuation (scores in the three sessions after the gain should be lower than the three before the gain). Over several years, clinical research has investigated SGs in several treatment contexts. They have been reported to be a ubiquitous phenomenon in psychotherapy that is significantly associated with treatment outcome (Shalom & Aderka, 2020). However, the mechanisms of its occurrence still remain poorly understood. A better understanding of the nature of this phenomenon and its relationship to change during treatment may help us improve psychological interventions.

Initially, and following Tang and DeRubeis (1999), SGs in cognitive behavioural therapy (CBT) were assumed to be the result of cognitive change. The authors found significant cognitive changes preceding SGs and postulated that cognitive changes triggered SGs (Tang & DeRubeis, 1999) This finding was replicated by Tang et al. (2005) and complemented by a more recent study showing decreases of negative appraisals shortly before SGs in patients with posttraumatic stress disorder(PTSD) (Wiedemann et al., 2020). However, several studies seem to contradict the cognitive change hypothesis of sudden gains (Bohn et al., 2013; Hofmann et al., 2006; Hunnicutt-Ferguson et al., 2012; Vincent & Norton, 2019; Vittengl et al., 2005). Importantly, if cognitive changes were the main mechanism leading to SGs, one would expect treatments like CBT to produce more SGs relative to treatments not directly focusing on modifying cognitions. Although an earlier meta-analysis had indicated higher rates of SGs in CBT compared to other treatments (Aderka et al., 2012), a more recent meta-analysis (Shalom & Aderka, 2020) showed that the type of treatment had no effect on the occurrence of SGs. Furthermore, SGs do not seem to be easily explainable by extratherapeutic factors (e.g. positive and negative life events) or as a result of regression to the mean (see Shalom et al., 2018, for an overview).

Importantly, some authors have argued that SGs are the result of random processes rather than significant mechanisms of change and therefore do not mark relevant points in the therapeutic process. With the use of Monte Carlo data simulation techniques, Thomas and Persons (2013) and Vittengl et al. (2015) concluded that the amount of SGs reported in the literature was comparable to the one in simulated data sets in which random errors were combined with either gradual linear or gradual curvilinear decreases of symptoms. Similarly, Persons (2022) argued as a result of simulation studies that SGs also occur under gradual patterns of change. Other approaches have proposed explanations for SGs in terms of extratherapeutic factors or regression to the mean, neither of which seems promising (see Shalom et al., 2018, for an overview).

Furthermore, SGs appear to have significant effects on other relevant factors including increases in therapeutic alliance (Lutz et al., 2013; Wucherpfennig et al., 2017), changes in coping skills (e.g. patients’ experiences of clarification of meaning and problem solving; Wucherpfennig et al., 2017), as well as changes in negative emotional states associated with PTSD (Kuck et al., 2023).

Recently, Aderka and Shalom (2021) proposed a revised theory of sudden gains based on the notion that SGs can be predicted by a measure of intra-individual variability (IV; Shalom et al., 2018; Shalom et al., 2020). The authors defined IV as the variability of a subject’s weekly symptom score, computed an index of this fluctuation and found it to correlate with the occurrence of SGs, independently of the change during treatment. However, this finding has not been replicated in another study and the described findings on intra-individual variability do not appear to generalize easily across treatments and disorders (Kuck et al., 2023). Explanations may be that more chronic psychopathology (as found in the study by Kuck et al., 2023) leads to less spontaneous variability or that there are other moderators to consider that exert an influence on variability.

In summary, there is no strong empirically supported explanation for the phenomenon of SGs or what predicts them (see Aderka & Shalom, 2021; Persons, 2022; Thomas & Persons, 2013; Vittengl et al., 2015). One possible way to advance the current understanding is to consider that a certain amount of SGs marks meaningful points of change during treatment while others might appear at random. This probabilistic focus promises to resolve some inconsistencies observed to date. In this context, reliable methods need to be tested and established to distinguish true and meaningful effects from randomly occurring SGs. In a recent approach by Lorenzo-Luaces et al. (2020), permutation tests were used to address this issue. By re-analysing previous data from cognitive and interpersonal therapy for depression, the authors showed that SGs occurred above and beyond chance level and that the relationship between SGs and better treatment outcomes was unlikely to occur by chance (Lemmens et al., 2016). The conducted permutation test is a robust non-parametric approach to evaluate the significance of SGs: Observed scores are shuffled (or ‘permuted’) within treatment sessions across 10,000 iterations while preserving original means and standard deviations. By comparing SG counts in the simulated datasets to those in the observed data, a p-value can be calculated that indicates the likelihood of the observed amount of SGs under the null hypothesis. For instance, in a hypothetical sample with 40% of subjects having an SG, if in 6,000 random re-samples among 10,000 permutations 40% of subjects show a sudden gain, the resulting p-value is .60 (6,000/10,000). This would suggest that SGs occurred at a chance level in the observed data. In sum, the permutation test provides us with a concise and statistically informed evaluation of an observed frequency of SGs in a treatment sample.

The findings by Lorenzo-Luaces et al. (2020) need to be replicated in populations with different disorders. As described, we still lack a coherent explanation for SGs and it remains to be examined whether the mechanisms of SGs are similar and the proposed method is applicable to individuals with other disorders or in a more naturalistic context. To the best of our knowledge, the work at hand is the first application of this form of permutation test in trauma-focused CBT and the first replication of the work by Lorenzo-Luaces et al. (2020). By demonstrating what proportion of SGs is to be expected by chance in the given data set, findings can guide further research on the use of adequate tests for the frequency of SGs. Importantly, this can shed light on the meaning of SGs for psychological treatments.

More recently, emphasis has also been placed on the distinction between ‘early’ and ‘late’ SGs in terms of when they occur in the therapeutic process. Sudden gains occurring earlier in treatment (mostly defined as occurring before the first third to half of a treatment) appear to be more closely connected to treatment outcomes than late gains (Busch et al., 2006) and subjects with early gains seem to have larger improvements and shorter treatment lengths (Lutz et al., 2013; Stiles et al., 2003). Interestingly, this parallels findings of early treatment responders having better treatment outcomes than late responders in general (Haas et al., 2002; Santor & Segal, 2001). This focus on the timing of SGs also appears promising for a more advanced understanding of the phenomenon and therefore constitutes a second aim of our study. Specifically, we expect that patients with early SGs show a lower symptom severity after the treatment. In addition, in an exploratory analysis, we will test whether SGs occurring in the trauma-focused phase of the treatment (which is expected to contain the specific mechanism of the treatment) are associated with a larger treatment effect.

2. Method

2.1. Participants

Participants were recruited from February 2014 to April 2016 at the outpatient centres of the University of Münster, Germany, and the Otto Selz Institute at the University of Mannheim, Germany. The data was collected for an effectiveness-study for trauma focused CBT in routine clinical care (for details see Krüger-Gottschalk et al., in preparation). A screening was conducted for all patients referred to the outpatient clinics and eligibility for the study was assessed via the Structured Clinical Interview for DSM-IV [SCID-IV, Wittchen et al., 1997]. Inclusion criteria were a primary diagnosis of PTSD (assessed with the CAPS-5; Weathers et al., 2018) and an age above 17 years. Exclusion criteria comprised acute suicidality, psychotic disorders, current substance dependence and a BMI < 17.5. Eligible patients provided written informed consent and the study was approved by the local ethics committees. In total, N = 85 patients (68 female; mean age = 35.84, SD = 12.85) were included, see Table 1 for demographic and clinical characteristics.

Table 1. Demographic and clinical characteristics of the sample at baseline.

	n (%)/M (SD)
Age (in years)	35.84 (12.85)
Gender
female	68 (80%)
male	17 (20%)
Relationship status
Not married, without partner	22 (25.88%)
Not married, with partner	28 (32.94%)
Married, living together	23 (27.06%)
Married, not living together	3 (3.53%)
Divorced	5 (5.88%)
Widowed	1 (1.18%)
Educational level
University degree	9 (10.59%)
High school^a	12 (14.11%)
Secondary school^b	52 (61.18%)
Primary school	3 (3.5%)
No degree	4 (4.71%)
Other	5 (5.89%)
Work status
Full-time job	30 (35.29%)
Part-time job	11 (12.94%)
Not working	12 (14.12%)
Unemployed	10 (11.76%)
Pensioner	5 (5.88%)
Other	11 (12.94%)
Type of trauma
interpersonal	54 (63.53%)
other	15 (17.65%)
Years since main trauma	10.49 (11.69)
Childhood abuse present	60 (70.59%)
CTQ subscales
emotional abuse	13.86 (6.53)
Emotional neglect	16.05 (6.63)
Physical abuse	9.61 (5.65)
Physical neglect	10.32 (5.07)
Sexual abuse	11.21 (7.26)
Number of axis-1 diagnoses (ICD-10)	2.04 (1.11)
Comorbidities
Any other axis-1 disorder	42 (49.41%)
Anxiety disorder	18 (28.18%)
Mood disorder	40 (47.09%)
Personality disorder	13 (15.29%)
History of substance dependence	7 (8.24%)
Current suicidal ideation	37 (43.53%)
Past suicide attempt	23 (27.06%)
Number of suicide attempts	0.55 (0.97)
Pre-treatment BDI-II score	28.49 (12.96)
Pre-treatment CAPS-5 sum score	37.81 (10.52)
Number of treatment sessions	37.42 (19.76)

Note. CTQ = Childhood trauma questionnaire; BDI-II = Beck Depression Inventory; CAPS-5 = Clinician-administered PTSD scale for DSM-5.

^a High school: 12–13 years of school education in the German school system.

^b Secondary school: 9–10 years of school education in the German school system.

2.2. Treatment

The applied treatment was a modularized trauma-focused cognitive behavioural therapy, consisting of a preparation phase, a trauma-focused phase (comprising imaginal exposure, discrimination training, changing dysfunctional appraisals) and a phase of reclaiming-your-life assignments and relapse prevention. Prior to treatment, screening sessions were held in which the structured clinical interviews were conducted (see Krüger-Gottschalk et al., in preparation, for more information). Treatment was conducted by registered CBT therapists or therapists in advanced postgraduate training under regular supervision. To reflect representative routine care conditions, treatment length was not determined a priori and therapists were allowed to apply the manual in a flexible and personalized way. On average, patients received M = 37.22 (SD = 20.15; range: 1–80) sessions (duration per session: 50 min) over a mean number of weeks of M = 56.04 (SD = 26.92), which is representative for the German health care system (granting up to 80 sessions for CBT treatment). A frequency of one session per week was aspired (mean length of between-session interval = 8.3 days, SD = 27.72) with longer intervals at the end of treatments to grant successful application of strategies for relapse prevention. Shortly before each treatment session, the process measures were completed. The main assessments were conducted before treatment, after completion and in follow-up measurements of 3, 6 and 12, and 24 months.

2.3. Measures

The main assessments were conducted prior to and following treatment and at follow-up assessments 3, 6, 12, and 24 months after the intervention. Before each weekly treatment session and at the main assessments, patients completed the self-report questionnaires (i.e. ‘process measures’).

The severity of PTSD symptoms in the last four weeks was measured with the German version (Müller-Engelmann et al., 2020) of the Clinician-Administered PTSD scale for DSM-5 (CAPS-5) at every main assessment. This structured clinical interview shows excellent reliability and validity and is widely used in PTSD-research (Weathers et al., 2018). Assessments were conducted by the therapists at pre, post- and the follow-up assessments.

At each treatment session and at all main assessments, the severity of post-traumatic symptoms was measured with the self-report PTSD Checklist for DSM-5 (PCL-5; Weathers et al., 2013; German version: Krüger-Gottschalk et al., 2017). In this 20-item self-report-instrument patients rate the distress caused by PTSD symptoms (from 0 – ‘not at all’ to 4 – ‘extremely’) in the past month (adapted here for the last seven days for the session-by-session assessments). The PCL-5 shows good psychometric properties and is a widely used measure in clinical studies of PTSD (Morrison et al., 2021).

For the assessment of depressive symptoms, the German version of the Beck Depression-Inventory-II (Beck et al., 1996; Hautzinger et al., 2006) was conducted, which has shown acceptable to strong psychometric properties (Kühner et al., 2007).

2.4. Statistical analysis

All data analysis was performed in RStudio (RStudio Team, 2020) using R (Version 4.1.1; R Core Team, 2020). Detection and processing of SGs was done with the R package ‘sudden gains’ (Wiedemann et al., 2019). Linear mixed models (LMM) were computed with the R Package lme (Pinheiro et al., 2017) employing the maximum-likelihood estimator. The significance criterion was set at α = .05 for all analyses.

Identification of sudden gains and sudden losses. For the detection of SGs and sudden losses in the weekly PCL-5-scores, the three criteria proposed by Tang and DeRubeis (1999) were considered and applied in analogy to Krüger et al. (2014) and Wiedemann et al. (2020). Firstly, a change between session N (pre-gain session) and N + 1 (post-gain session) had to be large in absolute terms. This was defined as exceeding the standard error of the difference $\mathrm{SED}=\sqrt{2\mathrm{SE}{M}^2}$ from the reliable change index (RCI; Jacobson & Truax, 1991), which is based on the standard error of measurement $\mathrm{SEM}=\mathrm{SD}\sqrt{1-r_{\mathrm{xx}}}$. Values for the standard deviation (SD = 19.99) and the re-test reliability (r_xx = .91) were drawn from a German clinical sample of trauma-exposed adults (Krüger-Gottschalk et al., 2017). This resulted in an SED of 8.5 points on the PCL-5, meaning a session-to-session improvement had to exceed this value to be considered as a sudden gain/ sudden loss. Secondly, the between session change had to be large relative to the symptom severity before the gain, defined as reaching at least 25% of the pregain symptom severity. Third, changes had to be stable, meaning being large relative to symptom fluctuation before and after the gain. This was fulfilled when the mean of the three PCL-5-scores before the gain was significantly smaller/greater than the mean of the three sessions following the gain/loss. Specifically, the difference between the means had to exceed the two-tailed t statistic (which used the pooled standard deviation of the three sessions before and after the gain). The critical values were adjusted for missing values. If only two sessions were available, these were taken for the computation.

As is common practice in the field, only the first sudden gain/loss per subject was included in the further analyses (on sg-prediction and the outcome-sg interaction). This ensures statistical independence of the SGs and allows the focus of analyses to be on the first major change in symptoms. The occurrence of SGs was exploratively examined in the entire sample and the frequency of SGs after session 20 will also be reported. Between-session intervals of more than 13 days were excluded for this analysis and only the first 20 sessions were analysed. This provided an adequate data quality for the permutation test and grants comparability to the method applied by Lorenzo-Luaces et al. (2020) who used data from a study with a protocol of 16–20 therapy sessions. Although analysing only the first 20 sessions in our sample (total range: 1–80) can be considered as a limitation, the mean number of conducted sessions was considerably lower than the possible maximum of sessions (M = 37.22; SD = 20.15), meaning that a considerable amount of therapeutic changes can be assumed to have taken place within the analysed sessions. The outcome was defined as the score at the end of treatment in the naturalistic setting.

Prediction of sudden gains, sudden losses and relationship with treatment outcome. For the prediction of sudden gains status (yes vs. no) in the weekly PCL-5-scores, univariate logistic regression analyses were conducted with demographic variables (age, gender) and pre-treatment clinical variables (total number of axis I diagnoses, comorbid anxiety disorder, comorbid mood disorder, months since main trauma, type of trauma, BDI-II score, comorbid personality disorder, history of substance dependence, suicidal ideation, past suicide attempt, childhood abuse, pre-treatment CAPS-5 score) as predictors.

To test whether subjects with SGs showed a lower symptom severity after the treatment, a linear mixed model [lmm] for the CAPS-5 sum scores was fitted with random intercepts for patients and time (pretreatment/ post-treatment), sudden gain status (yes/ no) and their interaction as fixed effects (model 1). To control for pre-treatment symptom severity, this was added to the model as a covariate. To gain insights on the timing of SGs with relation to the treatment phase, the first model was repeated considering only SGs occurring in the trauma focused phase (model 2). Another modified version of the first lmm was conducted to test whether early SGs differ from all SGs with respect to their effect on the treatment outcome (model 3). The model contained random intercepts for patients and fixed effects for an adopted sudden gain status variable (1 = early gain/ 0 = no early gain), the effect of time (pretreatment/ post-treatment) and their interaction on the CAPS-5-sum-scores. The median session of the occurrence of all SGs was computed and SGs were classified as early SGs if they occurred before this median session. The effect sizes for the effect of SGs on the treatment outcome were computed using the Cohen's d statistic (Cohen, 1988).

The described logistic regression models for the prediction of SGs and the lmm for the analysis of the relationship with the treatment outcome (model 1) were repeated in the same way for sudden losses.

Permutation tests. To assess whether the observed frequency of SGs in the weekly PCL-5-sum-scores in the current sample was higher than expected by chance, permutation tests were performed as suggested in Lorenzo-Luaces et al. (2020). The R-script used by the authors is accessible online (https://osf.io/d7rg2/). First, 10.000 data sets were created by randomly shuffling (or ‘randomizing’) the order of the original PCL-5 scores from the first 22 weeks of the treatment within each session (for exemplary symptom trajectories see figure B2). Thus, the data input for the permutation test yielded two additional sessions (21 and 22) to allow the detection of SGs within the first 20 sessions, as the third SG criterion (stability) always considers the two previous sessions before a potential SG. In this way, the SG detection in the original data (based on 20 sessions) was paralleled. With this permutation procedure, the means and standard deviations per session (across all participants) and the total symptom improvement across all participants was kept the same as in the original data. No autocorrelation between symptoms in the permuted data sets was expected, meaning that the session to session pattern of change was random for these scores.

Second, for each of the resulting 10.000 data sets, a sudden gains analysis was performed as described above for the original data. This gives a frequency distribution of SGs (under randomness) against which the found frequency of the real data set can be tested.

Third, the permutation test was conducted by comparing the SGs frequency of the original data against the obtained distribution from the permuted values. The achieved p-value is the proportion of the 10.000 permuted data sets, which contain equal or less SGs than the original data set.

3. Results

Information on the efficacy of the treatment and descriptive statistics will be reported in detail elsewhere (see Krüger-Gottschalk et al., in preparation).

Characteristics and timing of sudden gains and sudden losses. After having applied the criteria for sudden gains and losses, the analysis was based on 1616 between-session intervals for the whole treatment and 772 intervals within the first 20 treatment sessions (for complete data see Table 2). Twenty-four SGs were detected in total within the first 20 sessions.¹ Additional explorative analyses revealed five SGs and two sudden losses (both reversed later) during the screening sessions and five SGs after session twenty. There was no significant difference between study completers and dropouts concerning the number of experienced SGs, χ²(1) = 0.42, p = .52. The average symptom change around SGs can be found in Figure 1 (and see Figure B3 for example trajectories).

Figure 1. Average symptom change around sudden gains on the PCL-5. Note. The sudden gain takes place between the pregain session N and the postgain session N + 1.

Table 2. Characteristics of sudden gains and sudden losses.

	n (%)/M (SD)
Subjects suitable for sudden gains criteria^a	77 (91%)
Subjects with sudden gain	18 (23.38%)
Total number of sudden gains	24
Number of reversed sudden gains	8 (44.44%)
Magnitude of sudden gains^b	16.94 (8.22)
Number of early gains	8 (44.44%)
Mean and median session of sudden gains	9.41 (5.66), median = 10, range: 2–19
Timing of sudden gains according to treatment phase:
Preparation-phase	3 (16.67%)
Trauma-focused phase	9 (50%)
reclaiming-your-life assignments and relapse prevention	3 (16.67%)
Not assignable	3 (16.57%)
Start of trauma focused phase^c	11.52 (8.78), range: 1–39
Subjects with sudden loss^d	4 (5.19%)
Magnitude of sudden losses	19.25 (4.86)
Timing of sudden losses according to treatment phase
Preparation-phase	0
Trauma-focused phase	2 (11.11%)
reclaiming-your-life assignments and relapse prevention	2 (11.11%)
Mean and median session of sudden gains	11.5 (7.05), median = 11, range: 5–19

Note. ^asuitable for the application of the sudden gains criteria described in the method section and based on Tang and DeRubeis (1999).

^b in points of the PTSD Checklist for DSM-5.

^c values refer to the first session of trauma-focused work in each therapy.

^d two subjects with sudden loss were dropouts and two study completers. No subject experienced more than one loss.

Prediction of sudden gains, sudden losses and relationship with treatment outcome. None of the investigated predictors of sudden gain or sudden loss status (yes vs. no) reached significance in the univariate logistic regression models (all ps > .05; see Table A1). Subjects showed large effect sizes (according to Cohen, 1988) on the change of CAPS-5 scores from pre- to post-treatment (d = 2.23; see Figure 1 and Krüger-Gottschalk et al., in preparation, for details on the treatment’s effectiveness).

In all three linear mixed models investigating differences between subjects with and without SGs on the CAPS-5 scores, there was a significant main effect of time, with lower scores after the treatment (all ps < .05, for complete values see Table 3). In the first lmm (model 1), the interaction of time*sg-status was non-significant, F(1,53) = 0.36, d = 0.6, p = .55. This interaction remained non-significant in model 2, when considering only SGs occurring in the trauma-focused phase, F(1,53) = 0.03, d = 0.19, p = .86. However, when considering only early SGs (model 3), the time*sg-status-interaction was significant, F(1,53) = 4.97, d = 2.23, p = .03, showing that subjects with early SGs (M = 4.33, SD = 2.16) showed lower CAPS-5-scores at post-treatment than all other subjects (M = 12.37, SD = 11.76), t(45.43) = −4.3034, p < .001. To test for possible effects of SGs at later time points, model 1 was repeated with CAPS-5 scores at pre and follow-up 1. Again, the time*sg-status-interaction was non-significant, F(1,8) = 0.13, d = 0.36, p = .73. Similarly, the occurrence of sudden losses was not associated with treatment outcome, as the time*sl-status-interaction was non-significant, F(1,53) = 0.65, d = 0.81, p = .42. In Table 4 means and standard deviations before and after treatment are reported separately for subjects with and without SGs Figure 2.

Figure 2. Means of PTSD symptom scores in the ITT sample at pre, post, follow-up 1, and follow-up 2. Note. N = 85. Cohen’s d: d_CAPS-5 = 2.23, d_PCL-5= 1.86.

Table 3. Fixed effects of linear mixed models with time, sudden gains, and time*sudden gain interaction as predictors of CAPS-5 score.

	β [95% CI]	dfF	F	p
Model 1
(Intercept)	38.25 [35.37, 41.13]	1, 73	571.98	< .001***
Time	−24.98 [−28.11, −21.85]	1, 53	355.96	< .001***
Sudden gain	−1.38 [−7.50, 4.75]	1, 73	0.67	.41
Time*sg-status	−1.89 [−8.16, 4.38]	1, 53	0.36	.55
Model 2: sudden gains in phase 2
(Intercept)	38.61 [35.94, 41.28]	1, 73	584.89	< .001***
Time	−25.49 [−28.42, −22.56]	1, 53	355.69	< .001***
Sudden gain	−5.83 [−13.81, 2.14]	1, 73	2.52	.12
Time*sg-status	0.71 [−7.13, 8.55]	1, 53	.03	.86
Model 3: early sudden gains
(Intercept)	36.20 [29.70, 42.70]	1, 73	575.73	< .001***
Time	−22.25 [−28.51, −15.98]	1, 53	384.45	< .001***
Sudden gain	2.33 [−8.13, 12.79]	1, 73	1.19	.28
Time*sg-status	−12.20 [−22.49, −1.91]	1, 53	4.97	.03*

Note. Time (0 = pre, 1 = post). sg-status (indicating if subject experienced a sudden gain: 0 = no, 1 = yes). Random effect model 1: SD_Intercept = 8.12. Random effect model 2: SD_Intercept = 7.97. Random effect model 3: SD_Intercept = 6.16. *p < .05. ***p < .001.

Table 4. Mean (SD) of PTSD symptom severity of the intent-to-treat-sample on the CAPS-5 before and after treatment for patients with and without sudden gains.

	npre	Pre	npost	Post	nFU1	FU 1	nFU2	FU 2
Sg vs. rest
Sg = 0	59	38.33 (10.44)	42	12.21 (12.36)	9	9.33 (11.02)	23	11.87 (10.31)
Sg = 1	18	37.19 (11.25)	15	9.60 (8.22)	2	4.50 (2.12)	8	10.63 (8.43)
Sg phase 2 vs. rest
Sg = 0	68	38.68 (10.51)	48	12.19 (11.86)	9	9.33 (11.02)	24	11.96 (10.09)
Sg = 1	9	33.25 (10.28)	9	8.00 (8.22)	2	4.50 (2.12)	7	10.14 (8.99)
Early sg vs. rest
Sg = 0	69	38.01 (10.76)	51	12.37 (11.76)	9	9.33 (11.02)	27	11.74 (10.00)
Sg = 1	8	38.83 (8.70)	6	4.33 (2.16)	2	4.50 (2.12)	4	10.25 (8.88)

Note. N = available measurements for the respective groups. FU = Follow-up. CAPS-5: Clinician-Administered PTSD scale for DSM-5.

Permutation test of sudden gains. The application of the SG criteria on the 10.000 permuted data sets revealed an average of 17.75 (SD = 3.93; range: 4–35) SGs across an average of 16.37 (SD = 3.49; range: 4–31) patients. In a considerable amount of the permuted data sets (36.60%) equal or more patients than in the original data set experienced SGs (see Figure 3), suggesting that the number of patients with SGs in the original data is not greater than expected by chance (p = .37). Noteworthy, the total amount of SGs (across patients) in the original data showed a trend to be higher than chance level, as only 7.54% of the permuted data sets contain the same or a higher number of SGs (p = .08; see Figure 4). Three example trajectories each of the original and permuted data sets can be found in figure B2. Replicating the pattern in the original data, subjects with SGs (M = 11.55, SD = 3.04) did not differ from subjects without SGs (M = 11.52, SD = 0.79) with respect to the treatment outcome on the CAPS-5 (t(11348) = −0.91, p = .36).

Figure 3. Distribution of number of patients with sudden gains in 10,000 permuted datasets with randomized PCL-5 scores within sessions. Note. Black line: Mode of the distribution. Red dotted line: Number of patients with sudden gains in the original dataset.

Figure 4. Distribution of total numbers of sudden gains in 10,000 permuted datasets with randomized PCL-5 scores within session. Note. Black line: Mode of the distribution. Red dotted line: Number of sudden gains in the original dataset.

4. Discussion

We examined SGs in routine clinical care in a PTSD sample and provided an example of a successful application of a permutation test to evaluate the frequency of SGs. Less than one fourth of the subjects reported a sudden gain in the examined treatment period. As judged by the permutation test, this frequency was not higher than would be expected by chance.

A total of 29 SGs were reported by 18 participants (out of n = 85), with five SGs after session 20. The majority of SGs occurred during the phase of active trauma processing. Together with other evidence on the timing of SGs in relation to treatment manuals (Kuck et al., 2023), this suggests that SGs are connected to specific treatment mechanisms. In addition, no predictors of SGs or sudden losses were identified. This finding is in line with the literature on SGs (see Shalom & Aderka, 2020) and suggests that SGs are not connected in a simple fashion to pre-treatment patient characteristics.

Importantly, as judged by the permutation test the number of patients with SGs was not higher than expected by chance, while we found a trend for significance for the sum of SGs observed across participants. This result has to be integrated with the finding of an increased incidence of SGs during the trauma-focused treatment phase (vs. in the phases of preparation and reclaiming-your-life assignments) and findings from other studies (e.g. Kuck et al., 2023; Wiedemann et al., 2020), which connect SGs to specific behaviour change processes. We posit that during behaviour change processes, a certain amount of symptom reduction is categorized as SGs, with a portion of these categorizations occurring randomly.

Other possible explanations for the results of the permutation test are that the data or the methods used in our study differed significantly from other trials (resulting in a lower frequency of SGs or different mechanisms leading to them). Several aspects might have contributed to the relatively low frequency of SGs in our sample: The frequency of sudden changes in routine care settings has been found to be typically lower (around 23%) than in more controlled RCTs (around 40%, see Shalom & Aderka, 2020). Thus, although the number of SGs found in the current study deviates from that in earlier RCTs, it is within the range of what is typically found in routine clinical care. Differences between the study designs (RCT vs. more pragmatic studies) include less standardization, longer treatments and more individualized therapies as well as higher attention to comorbid disorders in routine care. Further, the intervals between sessions also show a higher variability, which might have affected the sg frequency. Notably, the found frequency in the current study still lies within the range of 14.30–62.20% found across studies by Shalom and Aderka (2020). Interestingly, the data from our sample resemble the findings on interpersonal therapy (IPT) in Lorenzo-Luaces et al. (2020), where the frequency was also not higher than expected by chance. Another possible explanation is that the patterns of change or mechanisms in the occurrence of SGs differ between disorders or treatments. Of note, the permutation test represents a very conservative test for the detection of SGs and it still remains open, whether it is easily applicable to naturalististic samples. We tested this method for the first time on trauma-focused CBT for patients with PTSD (vs. CT and IPT for depression in Lorenzo-Luaces et al., 2020), which means that replication with other clinical samples or more controlled study designs (e.g. in RCTs) is warranted.

Contrary to a larger body of research (Shalom & Aderka, 2020), we found no interaction of sudden gain status with the treatment outcome, which may be due to the specific study design: The treatment manual allowed therapists, in consultation with the supervisor, to switch between therapy phases in order to adapt to the patient's current symptoms. This means that both sudden gainers and non-sudden gainers received a well-matched and individualized treatment (also reflected in the large effect size) which might have reduced the variability of usual symptom trajectories and hereby influenced the association which is usually found. Additionally, the treatment length was not determined a priori, leading to good outcomes also for the non-sg-subjects. Together with the above-mentioned low frequency of SGs might be responsible for a missing association. Note, however, that some other studies also have found no relationship between SGs and treatment outcomes (Aderka et al., 2012). Furthermore, lack of statistical power may explain the lack of association between SG and outcome and the failure to predict SGs in our data.

Finally, a significant association between SGs and treatment outcome was found for early SGs in our sample. To date, differences between late and early gains are not well understood yet, and hence further research to understand their implications is required. Importantly, a more consistent operationalization of ‘early’ and ‘late gains’ is needed. As for the timing of SGs, in our sample, the majority of SGs occurred relatively late in the therapeutic process (median session: 10, vs. around 5 in most studies, see Aderka et al., 2012). We attribute this discrepancy to the fact that RCTs often implement treatment specific interventions earlier in treatment than in our study. In contrast, in our study the most potent (trauma-focused) intervention occured on average in later sessions, starting around session 11.

The primary strength of our study is that it was the first study to date to provide an example of a permutation test on SGs in routine clinical care. Findings inform the field on characteristics of SGs in this setting and provide an example for evaluating whether an observed frequency of SGs is higher than expected by chance. The naturalistic design results in a high external validity of the findings and allows conclusions to be drawn for applied psychological treatments. Yet, the naturalistic design also represents a potential limitation of the work at hand. Particularly, the flexibility of the therapists to being able to apply important therapeutic tools at different times during treatment makes the investigation of sg more difficult. Additionally, the generalization of our findings to other countries could also be questioned, as in the German health care system a larger number of therapy sessions is granted and the length of therapies was not determined a priori. Accordingly, the number of sessions might have been higher than in many other countries. The permutation test also has important limitations. The analysis is time-consuming and requires more computing power than other methods currently used in the field (see Lorenzo-Luaces et al., 2020). Further, the method does not explicitly account for temporal dependencies or autocorrelation in investigated time-series data. On the other hand, it involves fewer assumptions than other methods (see Berry et al., 2011).

The results of this study should be interpreted with caution when drawing clinical implications. However, our findings add to the body of literature indicating that SGs are a ubiquitous phenomenon and they might be connected to the effective mechanisms of treatments when happening early in treatment. Further, the absence of a correlation between sudden losses and therapy outcomes indicates that therapists should not be concerned by a temporary worsening of symptoms during treatment. Finally, the low number of sudden losses (around 5%) and the fact that they are mostly reversed during therapy encourages an optimistic approach to the treatment manual used.

Altogether, the approach we applied in this study seems promising for enhancing our understanding of the phenomenon of SGs. Future studies should examine whether the permutation test yields different results when applied in other forms of treatment and for other psychological disorders. Our results provide evidence for the practicality of permutation tests in the field and show that caution should be exercised in interpreting SGs, as a significant proportion of findings could be due to chance. Future research on the differential effects of early and late SGs in different clinical contexts is needed.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

We are unable to share the data publicly due to restrictions in the informed consent and ethical approval obtained for this study, which did not include the option of making the anonymised data publicly available.

Notes

1 A separate analysis of the SG-criteria showed that when only applying criterion 1 (and not two and three), 48 subjects showed a SG and when only applying criterion 1 together with criterion two or criterion three separately, 22 subjects showed a SG respectively.

Appendix

Appendix A

Table A1. Logistic regression analysis with demographic and baseline variables as predictors for sudden gain status (yes vs. no).

Predictor	OR [95% CI]	p
Demographic characteristics
Age	1.00 [0.96, 1.04]	.90
Sex	0.87 [0.18, 3.25]	.85
Baseline psychopathology
Total number of axis 1 diagnoses	0.98 [0.60, 1.57]	.93
Comorbid anxiety	0.92 [0.23, 3.07]	.89
Comorbid mood disorder	0.72 [0.24, 2.08]	.55
Months since main trauma	1.00 [1.00, 1.01]	.09
Type of trauma	1.43 [0.28, 6.06]	.64
Pre BDI	1.00 [0.96, 1.04]	.88
Comorbid personality disorder	0.98 [0.20, 3.71]	.98
History of substance dependence	0.51 [0.03, 3.35]	.56
Suicidal ideation	0.65 [0.21, 1.92]	.43
Past suicide attempt	1.21 [0.32, 4.23]	.76
Childhood abuse	1.22 [0.32, 5.97]	.78
Pre CAPS score	0.99 [0.94, 1.04]	.70

Note. A logistic regression model was calculated for each predictor variable separately. Sudden gains: 0 = no, 1 = yes. Sex: 0 = female, 1 = male. Comorbid anxiety: 0 = no, 1 = yes. Comorbid mood disorder: 0 = no, 1 = yes. Type of trauma: 0 = interpersonal, 1 = other. Comorbid personality disorder: 0 = no, 1 = yes. History of substance dependence: 0 = no, 1 = yes. Suicidal ideation: 0 = no, 1 = yes. Past suicide attempt: 0 = no, 1 = yes. Childhood abuse: 0 = no, 1 = yes.

Appendix B

Figure B1. Timing of sudden gains: distribution of pregain sessions.

Figure B2. PTSD symptom trajectories (PCL-5) of three example participants in the original and permuted data. Note. Pregain sessions are accentuated as squares.

Figure B3. Distribution of missing PCL-5 values within the first 20 therapy sessions for six example PTSD symptom trajectories. Note. Red bars represent missing PCL-5 values. The upper row shows examples for patients without sudden gains whereas in the lower row, examples for patients with sudden gains are presented.