Nav: Home

The big problem of small data: A new approach

October 18, 2018

Big Data is all the rage today, but Small Data matters too! Drawing reliable conclusions from small datasets, like those from clinical trials for rare diseases or in studies of endangered species, remains one of the trickiest obstacles in statistics. Now, Cold Spring Harbor Laboratory (CSHL) researchers have developed a new way to analyze small data, one inspired by advanced methods in theoretical physics, but available as easy-to-use software.

"Dealing with small datasets is a fundamental part of doing science," CSHL Assistant Professor Justin Kinney explained. The challenge is that, with very little data, it's not only hard to come to a conclusion; it's also hard to determine how certain your conclusions are.

"It's important to not only produce the best guess for what's going on, but also to say, 'This guess is probably correct,'" said Kinney.

A good example is clinical drug trials.

"When each data point is a patient, you will always be dealing with small datasets, and for very good reasons," he said. "You don't want to test a treatment on more people than you have to before determining if the drug is safe and effective. It's really important to be able to make these decisions with as little data as possible."

Quantifying that certainty has been difficult because of the assumptions that common statistical methods make. These assumptions were necessary back when standard methods were developed, before the computer age. But these approximations, Kinney notes, "can be catastrophic" on small datasets.

Now, Kinney's lab has crafted a modern computational approach called Density Estimation using Field Theory, or DEFT, that fixes these shortcomings. DEFT is freely available via an open source package called SUFTware.

In their recent paper, published in Physical Review Letters, Kinney's lab demonstrates DEFT on two datasets: national health statistics compiled by the World Health Organization, and traces of subatomic particles used by physicists at the Large Hadron Collider to reveal the existence of the Higgs boson particle.

Kinney says that being able to apply DEFT to such drastically diverse "real-world" situations --despite its computations being inspired by theoretical physics--is what makes the new approach so powerful.

"Flexibility is a really good thing... We're now adapting DEFT to problems in survival analysis, the type of statistics used in clinical trials," Kinney said. "Those new capabilities are going to be added to SUFTware as we continue developing this new approach to statistics."
-end-


Cold Spring Harbor Laboratory

Related Clinical Trials Articles:

Review evaluates how AI could boost the success of clinical trials
In a review publishing July 17, 2019 in the journal Trends in Pharmacological Sciences, researchers examined how artificial intelligence (AI) could affect drug development in the coming decade.
Kidney patients are neglected in clinical trials
The exclusion of patients with kidney diseases from clinical trials remains an unsolved problem that hinders optimal care of these patients.
Clinical trials beginning for possible preeclampsia treatment
For over 20 years, a team of researchers at Lund University has worked on developing a drug against preeclampsia -- a serious disorder which annually affects around 9 million pregnant women worldwide and is one of the main causes of death in both mothers and unborn babies.
Underenrollment in clinical trials: Patients not the problem
The authors of the study published this month in the Journal of Clinical Oncology investigated why many cancer clinical trials fail to enroll enough patients.
When designing clinical trials for huntington's disease, first ask the experts
Progress in understanding the genetic mutation responsible for Huntington's disease (HD) and at least some molecular underpinnings of the disease has resulted in a new era of clinical testing of potential treatments.
New ALS therapy in clinical trials
New research led by Washington University School of Medicine in St.
Telemedicine helps improve participation in clinical trials
Videos and creative uses of other visuals provide a novel way to obtain informed consent during clinical trials to improve participants' understanding and retention of trial information, according to a study by Nemours Children's Health System presented at the American Thoracic Society (ATS) Annual Conference.
Not enough women included in some heart disease clinical trials
Women are underrepresented in clinical trials for heart failure, coronary artery disease and acute coronary syndrome but proportionately or overrepresented in trials for hypertension, atrial fibrillation and pulmonary arterial hypertension, when compared to incidence or prevalence of women within each disease population, according to a study in the Journal of the American College of Cardiology.
BU: Obese patients underrepresented in cancer clinical trials
A new review by Boston University School of Public Health researchers found that less than one-fifth of participants in cancer-related clinical trials are obese.
Are women really under-represented in clinical trials?
Several studies have reported a lack of gender diversity in clinical trials, with trials including mostly adult males; however, a recent review of publicly available registration data of clinical trials at the US Food and Drug Administration for the most frequently prescribed drug classes found no evidence of any systemic significant under-representation of women.
More Clinical Trials News and Clinical Trials Current Events

Top Science Podcasts

We have hand picked the top science podcasts of 2019.
Now Playing: TED Radio Hour

In & Out Of Love
We think of love as a mysterious, unknowable force. Something that happens to us. But what if we could control it? This hour, TED speakers on whether we can decide to fall in — and out of — love. Guests include writer Mandy Len Catron, biological anthropologist Helen Fisher, musician Dessa, One Love CEO Katie Hood, and psychologist Guy Winch.
Now Playing: Science for the People

#543 Give a Nerd a Gift
Yup, you guessed it... it's Science for the People's annual holiday episode that helps you figure out what sciency books and gifts to get that special nerd on your list. Or maybe you're looking to build up your reading list for the holiday break and a geeky Christmas sweater to wear to an upcoming party. Returning are pop-science power-readers John Dupuis and Joanne Manaster to dish on the best science books they read this past year. And Rachelle Saunders and Bethany Brookshire squee in delight over some truly delightful science-themed non-book objects for those whose bookshelves are already full. Since...
Now Playing: Radiolab

An Announcement from Radiolab