Public opinion polls are often considered “the will of the people,” but a new study on the role of polls in South Korea shows that they may not always be that transparent.

“Using polls to gauge what people think about politics is not as simple as it sounds, as there are multiple mediating factors between what people think and how their views may be represented in the media,” says co-author Sunmin Kim, an assistant professor of sociology.

“Our research shows more broadly how in a democracy, measuring what people think or want can be a highly haphazard and unpredictable process based on the ways public opinion poll data are collected and reported.”

The findings are published in Politics & Society.

To examine the relationship between public opinion and polling, the research team chose to focus on South Korea because after the military dictatorship in the 1970s and 1980s, the country transitioned in a generation to a stable liberal democracy, where polling has become an integral part of the political fabric.

In addition to forecasting elections, polls are sometimes used to select the candidates for national primary elections. For example, during the 2002 presidential campaign, following a televised debate between two opposition candidates, Roh Moo-hyun of the Millennium Democratic Party and Chung Mong-joon, who ran as an independent, formed a coalition, with Roh as the candidate on the basis of winning a poll. He wound up defeating another candidate and winning the presidency.

In spring 2014, Kim and co-author Taeku Lee, the Bae Family Professor of Government at Harvard University, traveled twice to South Korea to interview 23 pollsters, journalists, academics, and campaign experts. They asked about the public perception of polls, the quality of the data, media coverage of poll results, and the relationship between polls and democracy in the country.

“There’s a contradiction between the ideal of polling and its practice,” says Kim. “The reality is that there’s all kinds of noise, including bad behavior, and opportunism that comes into play.”

Through this qualitative study in South Korea, the results demonstrate how the process of polling has its weaknesses, which pertain to technology, methodology, intentions, and reporting of data.

First, technology may affect a polling sample. Although South Korea boasts a 95% to 98% cell phone coverage rate, the interviewees caution that conducting polls by cell phone may result in overrepresenting young people.

Second, the way a poll is administered and whether it is automated or conducted in person can affect the response rate.

In South Korea, some political candidates hire polling firms to conduct polls, but in reality, they are hired to help with campaign messaging through the poll questions, such as by asking negative questions about a rival.

To obtain desirable data, pollsters may also distort the sample. If for example, a candidate is less popular among women, pollsters may systematically exclude women from the sample to get the numbers that they want.

The researchers also call attention to the role of “hit and run” operations by pollsters in local elections. A polling firm will set up operations in a county and obtain the results that a politician wants, and once they get paid for the work, they disappear, so that there’s no accountability. The pollsters may, however, show up in another county working under another firm name.

One of the other challenges with polls during an election season is the way that the results are reported by the media. According to the co-authors, the media often uses “horse-race polling” by focusing on who’s the frontrunner versus the underdog, which may help the leading candidate gain further momentum.

Consider a race that is especially close between two candidates: the stronger candidate is ahead by 5 percentage points and the margin of error is 4.5 points. Statistically speaking, the two candidates are effectively tied, but the media will focus instead on who is ahead and exclude the context.

“In the social sciences, public opinion polling has been widely accepted as a type of democratic process, but our study demonstrates how polls represent a social construction of public opinion, much of which is influenced by pollsters and their processes, and the media,” says Kim.