Conflation of errors in political polls

What is error?

Error is the difference between what we think is true and what is actually the case.  In statistical analysis, error can be classified into two general forms: random and non-random error.

Random error is sampling error, that is, the error that comes from taking a random sample from a population.  This error emerges because samples are not guaranteed to perfectly reflect the populations from which they are selected.  For illustration, imagine I want to know the average height of men in Canada.  I can’t measure the height of every man, it’s just not practical.  So I take a sample–say of 100 men.  Ideally, the sample is taken based on some sort of random process–like a computer program that randomly selects phone numbers from a list of all phone numbers.  A random selection process like this is most likely to cover the breadth of possible heights at their expected frequencies; some men are very tall, some men are very short but mostly are around the average.  However, it is possible that through this random sampling process that I select, just by chance, a sample that is taller than average.  This is possible because random samples are not guaranteed to look just like the populations they are drawn from.  This is analogous to the expected variability we’d get in a coin flipping experiment–we don’t expect heads to come up 50% of the time in every series of coin flips.  Instead, we expect a little of variability–a few more heads than tails (or vice versa) in a series of coin flips is not a surprise

Non-random error is a little more mysterious and has can come from many sources.  It can come from instrument errors, calculation errors, observer and participant biases, incorrect assumptions and a host of other problems that can occur throughout the research design process.  Non-random error is (generally) less of a problem in well designed experimental research–indeed, a properly designed randomized control trial has no non-random error and only random error.  Non-random error is a big problem for almost everything else humans do–including marketing products, climate models, sports analytics, crime prevention, medicine and public health, urban design…and on and on…

Random error has mathematical properties that allow us to understand it; it is a type of error that we can often estimate.  Statistical inference based on the calculation of ‘p-values’ is the conventional attempt to address random error, but it can’t really help us understand non-random error.


Polling errors are different…

The standard political poll says something like ‘in a survey of 10,000 likely voters, 32% of people plan to vote for candidate A’ with a margin of error of 1% 19 times out of 20.  In somewhat awkward statistical language this means that we expect the interval 31% to 33% to include the true % of people who will vote for candidate A, 19 times out of 20.  People reading the poll may think that it conveys a high degree of certainty–and that the small interval probably contains the true voter support and is therefore a good guess about the likely outcome of the election were it held today.  But unfortunately, many non-random errors are unaccounted for in political polling, and worse, may not be improved at all by taking large samples.  Just to name a few:

  1. People may be less willing to admit to voting for radical or unsavoury candidates in a phone or in-person poll (the ‘stealth voting‘ effect)
  2. People may change their votes at the polling both due to a desire to vote for the likely winner (the ‘bandwagon effect‘)
  3. Certain sampling methods may under-represent certain groups of voters
  4. People are influenced to change their voting behaviour based on polling results

All of these effects (and others) have been discussed in the political science literature on polling, so it’s not a new problem, and it should be of little surprise that so many polls seem to get election results wrong, even when taken shortly before election day.  The problem is that the reporting the magnitude of random error (as confidence intervals at a certain confidence level) makes it seem that polling error is known and small.  This ‘margin of error’ information gives too much authority to pollsters, and on occasion journalists have failed to dig deeply enough into the numbers to properly scrutinize the polling information.  Random error and statistical uncertainty are not the main problem with polling data; the main problem is more systematic, more unpredictable, and more difficult to explain.

The solution?

One approach is to include other error directly into the margin of error.  This isn’t easy, but some statistical theorists (like Bayesians) have ideas about how this could be done.  Generally speaking, it involves increasing the level of uncertainty in the margin of error based on what we know about the polling method and other factors.  If the current research shows that a certain data collection method has a had a very low success rate, this should be factored into the margin of error.  If one of the candidates has a poor public image, the results should be factored into the voting results.

It may also be possible to develop a more systematic understanding of how polls are wrong, and eventually improve their accuracy by making better adjustments to poll results. For example, if we know that polls systematically under-predict support for certain types of candidates (e.g., men, blowhards and idiots), then it may be possible to improve poll accuracy by boosting poll numbers for these candidates by a small amount.

These solutions take a lot of work, and while they have been an important part of modern election prediction, they still have a very mixed record of success.  It seems plausible that there will be some cyclic pattern to the accuracy of polls–periods of high accuracy followed by period of low accuracy–based on the ability of researchers to figure out the ways in which poll data are wrong over time.  Indeed, it seems plausible that if the effort to improve polling information continues, we may be able to expect better polling information in the near future.