Nav: Home

Study casts doubt on evidence for 'gold standard' psychological treatments

August 01, 2019

LAWRENCE -- A paper appearing today in a special edition of the Journal of Abnormal Psychology questions much of the statistical evidence underpinning therapies designated as "Empirically Supported Treatments," or ESTs, by Division 12 of the American Psychological Association.

For years, ESTs have represented a "gold standard" in research-supported psychotherapies for conditions like depression, schizophrenia, eating disorders, substance abuse, generalized anxiety and post-traumatic stress disorder. But recent concerns about the replicability of research findings in clinical psychology prompted the re-examination of their evidence.

The new study, led by researchers at the University of Kansas and University of Victoria, concluded that while underlying evidence for a small number of empirically supported treatments is strong, "power and replicability estimates were concerningly low across almost all ESTs, and individually, some ESTs scored poorly across multiple metrics."

"By some accounts, there are over 600 approaches to psychotherapy, and some are going to be more effective than others," said co-lead author Alexander Williams, program director of psychology and director of the Psychological Clinic for KU's Edwards Campus. "Since the 1970s, people have been trying to figure out which are most effective using clinical trials just like in medicine, where some subjects are assigned to a therapy and some to a control group. Division 12 of the APA has a list of therapies with strong scientific evidence from these trials, called ESTs. Ours is the first attempt anyone has made using this broad suite of statistical tools to evaluate the EST literature."

The researchers analyzed 78 ESTs with "strong" or "modest" research support, as determined by the APA's Society of Clinical Psychology Division 12, from more than 450 published articles. Four types of evidential value were assessed -- rates of misreported statistics, power, R-index and Bayes factors. Among the key conclusions:
  • 56% (44 of 78) of ESTs fared poorly across most metric scores.

  • 19% (15 of 78) of ESTs fared strongly across most metric scores.

  • 52% (26 of 50) of ESTs deemed by Division 12 of the APA as having Strong Research Support fared poorly across most metric scores.

  • 22% (11 of 50) of ESTs deemed by Division 12 of the APA as having Strong Research Support fared strongly across most metric scores.

  • 64% (18 of 28) of ESTs deemed by Division 12 of the APA as having Modest Research Support fared poorly across most metric scores.

  • 14% (4 of 28) of ESTs deemed by Division 12 of the APA as having Modest Research Support fared strongly across most metric scores.
"Our findings don't mean that therapy doesn't work, they don't mean that anything goes or everything is the same," said co-lead author John Sakaluk, assistant professor in the University of Victoria's Department of Psychology, who earned his doctorate at KU. "But based on this evidence, we don't know if most therapies designated as ESTs do actually have better science on their side compared to alternative, research-supported forms of therapy."

According to Williams, the field of clinical psychology may be ripe for a broad-scale reassessment of therapies that were thought to be supported by rigorous scientific evidence until now.

"Medical researchers coined a term called 'medical reversal,'" the KU researcher said. "Sometimes these are medical practices that doctors use across the country, but they are discontinued after it's found they don't work or aren't more effective than less-costly alternatives -- or they're actually harmful. Pending replications of our results, we may need broad systems-level psychotherapy reversals. Some of these ESTs are widely implemented in big systems like the Veterans Health Administration. If we find evidence for them isn't as strong as believed, it may be worth looking at. Let's say, hypothetically, there are two therapies for depression, and people have said, 'Well, Therapy A has stronger evidence for it than Therapy B.' But we know Therapy B works, too, and it's less costly. Today, if we find the evidence for Therapy A isn't actually stronger, it may be time to promote Therapy B."

Further, Williams advised clinicians and patients to continually evaluate progress in therapy and adjust therapeutic approaches based more on patient progress than research evidence of a given therapy's effectiveness.

"For clinicians and clients, this speaks to the importance of frequently assessing how well a client is doing in therapy," he said. "Routine outcome monitoring is always a good thing to be doing, but it may be a particularly good idea based on new evidence that we don't know if some therapies are effective. So, if I'm a patient, I want to assess how I'm doing -- and there are different measures for doing that. This study suggests it's even more important than previously believed."

For the research community, the authors recommended a reassessment of the size and power of clinical trials and more collaborations between labs to increase the precision of analyses, along with fresh approaches to how research is appraised, published and evaluated.

"One of the things that becomes really obvious when you look at the literature is researchers are collecting and analyzing their data in ways that are extremely flexible," Sakaluk said. "If you don't follow certain rules of statistical inference, you can inadvertently trick yourself into claiming effects that aren't really there. For EST research, it may become important to define in advance what researchers are going to do -- like how they'll analyze data -- and go on record in a way that restricts what they're going to do. This would coincide with a movement to encourage researchers to propose what they'd like to do and get reviewers and journal editors to weigh in before -- not after -- scientists do research, and to publish it irrespective of what they find."

Williams said studies supporting the power of clinical treatments should improve over time with more exacting approaches to statistical data.

"This is a system-level issue that will get better as our field begins to grapple with replication," he said. "We think you'll see improvement in study design going forward. There wasn't a fieldwide appreciation for these problems until a decade ago. It takes time for the field to improve. We think our results will complement ongoing efforts by Division 12 to increase the quality of EST research and evaluation."
-end-
Williams and Sakaluk's co-authors were Robyn Kilshaw of the University of Utah and Kathleen Teresa Rhyner of the Canandaigua VA Medical Center, the latter of whom also earned her doctorate at KU.

University of Kansas

Related Science Articles:

75 science societies urge the education department to base Title IX sexual harassment regulations on evidence and science
The American Educational Research Association (AERA) and the American Association for the Advancement of Science (AAAS) today led 75 scientific societies in submitting comments on the US Department of Education's proposed changes to Title IX regulations.
Science/Science Careers' survey ranks top biotech, biopharma, and pharma employers
The Science and Science Careers' 2018 annual Top Employers Survey polled employees in the biotechnology, biopharmaceutical, pharmaceutical, and related industries to determine the 20 best employers in these industries as well as their driving characteristics.
Science in the palm of your hand: How citizen science transforms passive learners
Citizen science projects can engage even children who previously were not interested in science.
Applied science may yield more translational research publications than basic science
While translational research can happen at any stage of the research process, a recent investigation of behavioral and social science research awards granted by the NIH between 2008 and 2014 revealed that applied science yielded a higher volume of translational research publications than basic science, according to a study published May 9, 2018 in the open-access journal PLOS ONE by Xueying Han from the Science and Technology Policy Institute, USA, and colleagues.
Prominent academics, including Salk's Thomas Albright, call for more science in forensic science
Six scientists who recently served on the National Commission on Forensic Science are calling on the scientific community at large to advocate for increased research and financial support of forensic science as well as the introduction of empirical testing requirements to ensure the validity of outcomes.
World Science Forum 2017 Jordan issues Science for Peace Declaration
On behalf of the coordinating organizations responsible for delivering the World Science Forum Jordan, the concluding Science for Peace Declaration issued at the Dead Sea represents a global call for action to science and society to build a future that promises greater equality, security and opportunity for all, and in which science plays an increasingly prominent role as an enabler of fair and sustainable development.
PETA science group promotes animal-free science at society of toxicology conference
The PETA International Science Consortium Ltd. is presenting two posters on animal-free methods for testing inhalation toxicity at the 56th annual Society of Toxicology (SOT) meeting March 12 to 16, 2017, in Baltimore, Maryland.
Citizen Science in the Digital Age: Rhetoric, Science and Public Engagement
James Wynn's timely investigation highlights scientific studies grounded in publicly gathered data and probes the rhetoric these studies employ.
Science/Science Careers' survey ranks top biotech, pharma, and biopharma employers
The Science and Science Careers' 2016 annual Top Employers Survey polled employees in the biotechnology, biopharmaceutical, pharmaceutical, and related industries to determine the 20 best employers in these industries as well as their driving characteristics.
Three natural science professors win TJ Park Science Fellowship
Professor Jung-Min Kee (Department of Chemistry, UNIST), Professor Kyudong Choi (Department of Mathematical Sciences, UNIST), and Professor Kwanpyo Kim (Department of Physics, UNIST) are the recipients of the Cheong-Am (TJ Park) Science Fellowship of the year 2016.
More Science News and Science Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Listen Again: Reinvention
Change is hard, but it's also an opportunity to discover and reimagine what you thought you knew. From our economy, to music, to even ourselves–this hour TED speakers explore the power of reinvention. Guests include OK Go lead singer Damian Kulash Jr., former college gymnastics coach Valorie Kondos Field, Stockton Mayor Michael Tubbs, and entrepreneur Nick Hanauer.
Now Playing: Science for the People

#562 Superbug to Bedside
By now we're all good and scared about antibiotic resistance, one of the many things coming to get us all. But there's good news, sort of. News antibiotics are coming out! How do they get tested? What does that kind of a trial look like and how does it happen? Host Bethany Brookeshire talks with Matt McCarthy, author of "Superbugs: The Race to Stop an Epidemic", about the ins and outs of testing a new antibiotic in the hospital.
Now Playing: Radiolab

Dispatch 6: Strange Times
Covid has disrupted the most basic routines of our days and nights. But in the middle of a conversation about how to fight the virus, we find a place impervious to the stalled plans and frenetic demands of the outside world. It's a very different kind of front line, where urgent work means moving slow, and time is marked out in tiny pre-planned steps. Then, on a walk through the woods, we consider how the tempo of our lives affects our minds and discover how the beats of biology shape our bodies. This episode was produced with help from Molly Webster and Tracie Hunte. Support Radiolab today at Radiolab.org/donate.