Making sense of surveys; when to trust polls

July 01, 2002

WASHINGTON, DC -- Being able to understand and trust surveys is critical to Americans becoming informed citizens, but mass media often report surveys without the requisite practical guidance on the context of the results or how to interpret and evaluate them. To help guide the public in comprehending the significance of surveys and questionnaires, sociologist Howard Schuman explains in plain language the nature, importance, and problematic aspects of surveys in the summer edition of Contexts magazine, the newest publication of the American Sociological Association.

Schuman, Professor and Research Scientist Emeritus at the University of Michigan, notes in his article, "Sense and Nonsense about Surveys," that surveys have appealed since ancient times to two human propensities: (1) Gathering information by asking questions, and (2) attempting to learn more about one's environment by examining a small part of it (i.e., sampling).

Through an illustrative example of the sampling conducted by the infamous Literary Digest poll prediction of the 1936 U.S. presidential election, Schuman demonstrates both the problems and progress made in survey methodology. On the basis of a very large sample--2 million completed and returned surveys out of 10 million distributed--the Literary Digest had incorrectly predicted that Franklin D. Roosevelt would lose decisively in the election.

"At the same time, pollster George Gallup, using many fewer cases but a better method, accurately predicted that Roosevelt would win. Gallup used quotas in choosing respondents in order to represent different economic strata, whereas the Digest had worked mainly from telephone and automobile ownership lists, which in 1936 were more likely to include wealthy people who were likely to be opposed to Roosevelt," explained Schuman. (There were other sources of bias as well.) As a result, the Digest disappeared from the scene, and Gallup was on his way to becoming a household name.

Schuman highlights two advances in survey methods since the 1930s and 1940s that allowed the modern survey to develop and flourish. "The first is the emergence of probability sampling, which is fundamental to social science research today and helps overcome the problem of biased sampling of respondents. The second relates to the development of greater precision in asking questions in surveys and in replicating and testing the validity of the questions themselves," said Schuman.

Obtaining a good sample depends on more than adhering to principles of good probability sampling (i.e., selecting respondents from the population in proportion to the degree to which they are representative of that population). Well-done surveys also depend on the quality of their implementation. Thus, when members of a population who are selected to be in the sample are not successfully interviewed or do not respond to specific questions, the omissions can seriously compromise the validity and reliability of the survey and are as important as the sample size. Interestingly, however, for mathematical reasons, reasonably accurate estimates can be obtained--from populations of very different sizes--with sample sizes of around 1,000, and only when extraordinary precision is needed do samples need to be much larger.

The "margin of error" plus/minus percentage figures that accompany most media reports of polls reflect the size of the final sample, but they do not reveal the sampling methods or the extent to which the targeted individuals or households were actually included in the final sample. For example, the percentage of people who refuse to take part in a survey is particularly important. In some federal surveys, the percentage is small, within the range of 5 to 10 percent. But even in the best non-government surveys, the refusal rate can reach 25 percent or more, and it can be far larger in the case of poorly executed surveys. The opinions of non-responders in a population may be very different from those of responders, thus their non-inclusion biases results.

Many survey reports are based on such poor sampling procedures that they do not deserve to be taken seriously. This is especially true of reports based on "focus groups," which offer human interest but are subject to vast amounts of error. Internet surveys also cannot represent the general population adequately at present, though this is an area where some serious attempts are being made to compensate for the inherent difficulties and inherent sampling bias.

Schuman also describes several techniques researchers use to improve survey questioning--such as asking several different types of questions about any important issue, varying the form, wording, and context of questions, and introducing comparisons into analyses. These techniques are utilized in order to overcome variations in response that are produced not by the responders actual opinion but by the peculiar form, wording, or frame of reference of questions in a survey. Using several examples from actual surveys, he shows how different ways of asking the same question can produce substantial differences in outcomes.

Despite the methodological advances, some issues still remain. For example, pollsters still face the task of interpreting the meaning of questions and the associations among questions, but this is true in all types of research including those using field observations.

Schuman states that surveys remain our best tool for learning about large populations. "In the end, however, with surveys, as with all research methods, there is no substitute for both care and intelligence in the way evidence is gathered and interpreted. What we learn about society is always mediated by the instruments we use, including our own eyes and ears," concluded Schuman.
The American Sociological Association, founded in 1905, is a non-profit membership association dedicated to serving sociologists in their work, advancing sociology as a science and profession, and promoting the contributions and use of sociology to society.

American Sociological Association

Related Mass Media Articles from Brightsurf:

Discovery of a new mass extinction
It's not often a new mass extinction is identified; after all, such events were so devastating they really stand out in the fossil record.

How vitamin C could help over 50s retain muscle mass
New research shows that vitamin C could help over 50s retain muscle mass in later life.

Proximity of hospitals to mass shootings in US
Nontrauma center hospitals were the nearest hospitals to most of the mass shootings (five or more people injured or killed by a gun) that happened in the US in 2019.

Keeping guns away from potential mass shooters
Researchers from Michigan State University measured the extent to which mass shootings are committed by domestic violence perpetrators, as well as identyifying how they illegally obtain guns, suggesting how firearm restrictions may prevent these tragedies.

Who is left behind in Mass Drug Administration?
Ensuring equity in the prevention of neglected tropical diseases (NTDs) is critical to reach NTD elimination goals as well as to inform Universal Health Coverage (UHC).

A mechanism capable of preserving muscle mass
By studying the young and aging muscles in mice, researchers from the Myology Research Center (Sorbonne Universite-Inserm) of the Institute of Myology identified a protein, CaVbeta1E that activates the factor GDF5.

Cynical social media voices can erode trust in news media
Amid rising concerns about low public trust in mainstream media institutions, a Rutgers study found that real-life and online social interactions can strongly influence a person's trust in newspaper, TV and online journalism -- but when it comes to online interactions, cynical views are the most influential.

Eyeballing a black hole's mass
There are no scales for weighing black holes. Yet astrophysicists from the Moscow Institute of Physics and Technology have devised a new way for indirectly measuring the mass of a black hole, while also confirming its existence.

Social media stress can lead to social media addiction
Social network users risk becoming more and more addicted to social media platforms even as they experience stress from their use.

'Fake news,' diminishing media trust and the role of social media
Exploring the perception of the 'fake news' phenomenon is critical to combating the ongoing global erosion of trust in the media according to a study co-authored by a University of Houston researcher.

Read More: Mass Media News and Mass Media Current Events is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to