All have won what? On the epistemic value of RCT based evidence in Evidence Based Treatments

Society for Psychotherapy Research (SPR)
24-26 September 2015, Klagenfurt, Austria

All have won what? On the epistemic value of RCT based evidence in Evidence Based Treatments

Aim “All are effective, and all must have prizes,” – so the dodo bird verdicts the statistical indifference between ‘evidence based’ types of psychotherapy. Evidence is understood as the result of randomized controlled (RCT) research. This ‘golden standard’ design requires samples that are homogeneous with regard to symptom(s) – following from the statistical assumptions in central tendency statistics. Symptom specific measures are used to select patients scoring above cut-off, resulting in samples characterized by tight and simple symptom patterns. Whereas the external validity of this procedure is heavily critiqued, I study the internal validity of a priori methodological assumptions in RCT efficacy research. In this paper, I address the clinical plausibility of sample selection by a quantified symptom measure for depression, the BDI-II, which was found to be a reliable screening measure for Major Depressive Disorder.

Method In a pilot study on our mixed method psychotherapy data from depressed adults, we used the BDI-II to select patients eligible to participate in a strict RCT design. For both the eligible and non-eligible sample, we scrutinized individual symptom patterns.

ResultsAnalysis showed a 60-40 ratio of depressed patients to be included or excluded, respectively. 75% of the ‘eligible’ sample still showed comorbidity, which would have been reason to exclude them from a RCT.

Discussion The extraordinary large exclusion percentage indeed inserts doubt on external and ecological validity of quantified eligibility screening in RCT efficacy research. More striking however is that a supposedly ‘eligible sample’ still harms the assumption of homogeneity. In this paper, I propose such invalidity of a priori methodological assumptions as an explanation for the dodo bird effect. I discuss the implications of these preliminary empirical results for the understanding of ‘evidence’ from RCTs and the epistemic value of Evidence Based Treatments.




Comments are closed.