Can polls be trusted?
Potential Problems with Opinion Polls (June 1, 2023)
Now that you understand how to read a poll, you can start to rely upon them. Not so fast! There can be a number of problems with opinion polls; some you may be able to see and others you won’t.
One problem with polls relates to how responses were collected. Ideally, it was based on a truly representative (random) sampling of the full target group. However, most opinion gathering techniques have built-in biases. Online polling biases towards the opinions of technologically inclined people. Land-line phone polls bias towards older folks who are more inclined to have a home phone. TV polls bias towards TV watchers, who are weighted towards older folks these days. Mail polls bias toward people willing to deal with such an antiquated and environmentally contentious process. On a more general basis, any kind of poll will bias towards people who are less time-pressured and have time to complete them. Basically, all approaches have some bias. If the bias is significant, hopefully the reported results will say the polling was 'not scientific.' In other words, you really can't get much of anything from them.
Another problem area is the order in which poll questions are asked. To illustrate, imagine a simple poll that asks two questions. One is on your general satisfaction level with the current government. The other asks about your view on the significance of a reported government scandal. You can easily surmise that the response to the first question above would be more critical if the other question is asked before it.
A third problem is that polls can contain leading questions. For example, compare “What is your view of the government's response to the flood?” to “What is your view of the government's response to one thousand people losing their homes to flooding and having nowhere else to live?” I think the responses would be significantly different.
A fourth problem relates to the timing of a poll. You know the expression “get ‘em while they’re hot.” Polls too can get to people when they’re hot. If a poll is taken immediately after a negative public event, you will get a strong dissenting response. If you let people cool off, the same questions will get a more reasoned response. The later poll will be more accurate, but the earlier one can be better used to sensationalize or manipulate.
And finally, a word about evil polls. Yes, polls can be evil. The most egregious are called push-polls. Legitimate polls seek to 'pull' people's views on an issue. Push-polls exist only to 'push' an opinion or doubt onto the poll respondent. An example would look like this. An organization fabricates a nasty rumour about a political opponent. A phone poll is run to ask folks whether it would change their likelihood of voting for the candidate if such a matter was to become widely known. The pollsters couldn't care less what the poll respondents say. They just want to get them talking and posting about the rumour, somewhat legitimized by the fact that there is polling being done. In fact, the nature of the poll question strongly suggests that it is the ‘scandal-ridden’ party that is running the poll. So it must be true!
A milder form of a push poll would be when a marketing company conducts opinion surveys on their client’s products. There is always the chance that the poll is just to get the word out about a new product and its fabulous improvements without sounding like a sales pitch.
Polls can be misleading; sometimes intentionally. This might be partly the reason that pollsters rank very low on the list of professions thought of positively by the public (2017, 34%, being third from the bottom). Another reason is likely that they are always interrupting us at the most inopportune times. A third reason may be some high-profile misses being made, notably in polling leading up to an election. Whatever the reasons, you have to love the irony that it took an opinion poll to tell us how we feel about pollsters.
And it will not be easy to tell a good poll from a bad one. That is why in Canada pollsters and journalists are required by law to report the following information for political polls during an election: who sponsored the survey; who conducted the poll; when the poll was conducted; the population from which the survey was drawn; how many persons were contacted; the survey’s margin of error; and if the survey was not conducted using recognized statistical methods2.
This information might help you see potential problems in political polling during an election. No similar requirements exist for all other polls. If a poll is important to you, go look for the information. A reputable pollster will make it easy to find. If it’s not there, beware.
____________________
1. Insight West, link no longer available
2. Elections Canada, Opinion Polls Must Comply with Election Law
Rules Governing Publishing Poll Results Set Out in Canada Elections Act,
https://www.elections.ca/content.aspx?section=med&document=mar2911&dir=pre&lang=e