What is that gobbledygook reported with polls?

How Opinion Polls Work (June 1, 2023)

Surveys can be used effectively to determine the behaviours and opinions of a defined group of people, all of whom will be questioned.  Polling is different in that it seeks to obtain information from a representative sample of all members of a group, and then infer these results to the whole group.  Polling can be very effective if done right.  Churches may from time-to-time come across poll results that they wish to use in planning and program development.  But they need to understand them first.

The primary consideration with respect to polls lies in a mathematical fact of using sampling techniques.  All results must be interpreted in terms of a margin of error and a degree of certainty.  Without such information, results can be misinterpreted or misapplied. 

Accordingly, you see poll results accompanied with a statement such as 'plus or minus (±) four points, nineteen times out of twenty.'  It sounds cryptic, but it is quite straight-forward.  For example, if the poll reports public agreement of 60% on any particular question, it means that 95% of the time (19 divided by 20) identical polls would yield results on this question ranging from 56% to 64%.  Five percent of the time, such a poll would yield results that are lower or higher than this range.

The key determinant of these statistics is the size of the sample used.  The larger the sample, the more narrow the error range and/or greater the certainty factor.  Polls tend to try to stay the same in the degree of certainty (usually 95%), so sample size will just about always play out in the margin of error.  It takes longer and costs more to poll larger groups of people so there is always pressure to keep a sample as small as possible.  Knowing this, you can use the error range to judge for yourself whether the poll was large enough to yield the preciseness of information you need.  

Further, the margin of error should be considered a little more carefully when a poll reports on multiple opinions on any particular matter that add up to 100%.  For example, a poll could report that 46% indicate they will vote Liberal in the next election and 54% say they will vote for Conservative.  (An over-simplified example to make my point).  When we apply the margin of error in such a case, we can see any error in one figure affects the other factor equally.  So a 4% margin of error means it is possible that 50% could vote Liberal, and then 50% will vote Conservative, still adding to 100%.  For this reason, pollsters and journalists will often label close polling results as “a statistical tie.”

Another consideration in all polls is the people who say they are undecided or not willing to give a response.  If a poll is conducted on an issue that is too recent (in today’s news) or will not concern the respondent for a while (like an election several months away), the number of opinions collected will fall quite short of the number of people contacted.  It is always difficult to assess these polls as it involves assumptions about the undecideds.   A natural inclination is to feel that this group will split the same way as the people who gave an opinion.  However, this will not always be the case.  For example. in politics, undecideds tend to lean towards the party currently in power.  

And finally, a word about percentage points.  Pollsters and the media often refer to percentage points when talking about the difference in the rate of agreement between one question and another.  For example, if 40% agree to option A and 46% agree with option B, then the two are separated by 6 percentage points.  The key is not to confuse this with percentage difference.  The percentage difference between these two numbers is 15%, calculated by taking the difference (6), dividing it with the first figure mentioned (40), and multiplying this by 100.

For these reasons, polls are not usually a precision tool.  They are more like a chainsaw than a hand saw.  They are useful for determining trends in opinions over time and for establishing the existence of large disparities.  However, if a public issue is looking like a neck-and-neck race then polls generally cannot be relied upon to tell you who is going to come out on top.