Bad Statistical Methods Nix New Discoveries

March 09, 1998

Obsolete Methods Cause Researchers To Miss Significant Discoveries, USC Psychologist Says

In a provocative new paper, a University of Southern California quantitative psychologist says many psychology researchers are using obsolete statistical techniques and as a result are missing important discoveries.

"Psychology journals are littered with nonsignificant results that would have been significant if a more modern method had been used," says Rand R. Wilcox, Ph.D., writing in the March issue of American Psychologist.

Dr. Wilcox, a professor of psychology, a statistician and the author of several books on statistics and social science research, suggests that psychology researchers should revisit many older studies and re-analyze the data using modern computational techniques. Furthermore, he says, psychology education must be upgraded so that psychologists are aware of the new techniques.

"It's in everybody's best interest to take advantage of modern technology," says Wilcox, who teaches statistical analysis to psychology majors in the USC College of Letters, Arts and Sciences. "Computers today can easily do calculations that were impossible 50 or even five years ago."

As cutting-edge statisticians have developed increasingly sophisticated techniques for handling data, Wilcox says, "there's an ever-widening gap between modern statistical methods and the techniques psychologists are continuing to use."

Typically, psychology studies involve comparisons of two or more groups of subjects, with researchers collecting data on individuals and looking at the variables, Wilcox explains. The classic analytical tool familiar to every psychologist is the T test, in which the average, or mean, of each group is compared to the others'.

The T test is easy to understand and easy to calculate, and, says Wilcox, it can easily lead to the wrong conclusion. For example, if two groups of people each have an average IQ of 100, you might conclude from the T test that there is no significant intelligence difference between the two groups.

But what if every individual in the first group had an IQ very close to 100 while the second group included some very intelligent people with IQs of 125 or more and some very unintelligent people with IQs of 75 or less? Then the two groups can have differences that the T test would not reveal.

Although this may seem a gross simplification, many researchers have essentially made this mistake with the T test, Wilcox says.

When researchers collect data for a study, they usually find some values that are unusually large or small. In the example above, one IQ might have been as high as 180 and one as low as 50. Such data are called outliers, and while they may be interesting by themselves, they wreak havoc with standard statistical methods.

Leave them in, and the outliers can skew a sample. Leave them out and apply standard methods to the data that remain -- a common strategy -- and serious technical problems may result. Outliers can be handled most effectively, Wilcox says, by carefully trimming the outliers and by applying theoretically correct statistical methods.

Wilcox says researchers have long ignored slight differences from a normal curve because they cannot handle them mathematically. Again, consider IQ as an example. Researchers assume that if you graphically plotted the IQ scores of a random sample of people, their scores would probably fall into an almost perfect bell-shaped curve. Most people are of average intelligence and their scores would be plotted at the top of the bell. The less intelligence individuals had, the farther down on the left-hand descending side of the bell their scores would fall; while the more intelligence they had, the farther down their scores would be on the right-hand descending side.

For a long time, psychologists have known that the IQ scores of a small number of extremely intelligent people can skew the upper-right side of the bell curve by pulling it out to form a little tail.

Many psychology researchers have pretended this slight distortion just doesn't occur, but it can make a real difference when looking for relationships between two groups, Wilcox points out.

In a famous study done 30 years ago, a psychology researcher compared the intelligence of two groups of children. The teachers of the children in the first group were told they could expect their students to do well on IQ tests. The teachers of the second group were told nothing about their students' test- score potential.

For decades, psychology researchers have questioned whether the children for whom positive expectancies had been suggested actually scored higher on subsequent IQ tests. Differences between the two groups were not easily studied or understood by using conventional methods. "Using modern statistical analysis and taking pretest scores into account, there is no compelling reason to believe that expectancies influence IQ test scores," Wilcox says.

Wilcox believes the problem of using old computational technique is not limited to the field of psychology but probably pervades many other research fields as well. Nor is the problem confined to research alone.

"Some of the most popular statistical software packages available today are four generations out of date," he says.


University of Southern California

Related Intelligence Articles from Brightsurf:

Human intelligence just got less mysterious says Leicester neuroscientist
NEUROSCIENCE EXPERTS from the University of Leicester have released research that breaks with the past fifty years of neuroscientific opinion, arguing that the way we store memories is key to making human intelligence superior to that of animals.

Artificial intelligence (AI)-aided disease prediction
Artificial Intelligence (AI)-aided Disease Prediction Announcing a new article publication for BIO Integration journal.

New roles for clinicians in the age of artificial intelligence
New Roles for Clinicians in the Age of Artificial Intelligence Announcing a new article publication for BIO Integration journal.

Classifying galaxies with artificial intelligence
Astronomers have applied artificial intelligence (AI) to ultra-wide field-of-view images of the distant Universe captured by the Subaru Telescope, and have achieved a very high accuracy for finding and classifying spiral galaxies in those images.

Using artificial intelligence to smell the roses
A pair of researchers at the University of California, Riverside, has used machine learning to understand what a chemical smells like -- a research breakthrough with potential applications in the food flavor and fragrance industries.

Artificial intelligence can help some businesses but may not work for others
The temptation for businesses to use artificial intelligence and other technology to improve performance, drive down labor costs, and better the bottom line is understandable.

Artificial intelligence for very young brains
Montreal's CHU Sainte-Justine children's hospital and the √ČTS engineering school pool their expertise to develop an innovative new technology for the segmentation of neonatal brain images.

Putting artificial intelligence to work in the lab
An Australian-German collaboration has demonstrated fully-autonomous SPM operation, applying artificial intelligence and deep learning to remove the need for constant human supervision.

Composing new proteins with artificial intelligence
Scientists have long studied how to improve proteins or design new ones.

Artificial intelligence and family medicine: Better together
Researcher at the University of Houston are encouraging family medicine physicians to actively engage in the development and evolution of artificial intelligence to open new horizons that make AI more effective, equitable and pervasive.

Read More: Intelligence News and Intelligence Current Events is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to