Margin of Error
In the world of survey research, there are some pretty bizarre debates going on. The latest controversy is about whether one should report a margin of error for an online survey. I can honestly say that most folks have never lost a minute of sleep pondering this. That being said, it is a very important topic for those who conduct survey research. Furthermore, the starting point of the discussion is that almost everyone who is a part of the debate acknowledges that it is completely inappropriate to calculate a margin of error for an online survey from a theoretical basis. Samples for online surveys are referred to as “model based” and, in particular, are not probability samples. The ability to calculate a margin of error for a survey sample comes from the distributional characteristics of a probability sample.
Here is the surprising part with respect to almost all of survey and market research when it comes to margin of error: It is all a sham! How can this be? If you look closely at each political poll that comes out or at more research presented in the media, there is a margin of error attached to it. It provides comfort in that it gives a sense to the reader of how far off the actual value may be from the survey value. But for the same reason that it cannot be theoretically calculated for model-based and, hence, online samples, other types of samples that might be called “probability samples” by the layman do not meet the conditions necessary to report a margin of error.
Let’s take survey research conducted via telephone as an example. When I started in the field in the late 1990s, we had to load seven or eight pieces of sample for each interview we wanted to conduct by telephone. There are a variety of reasons for needing more sample than interviews required. In the late 1990s, the primary reason for needing so much sample was that no one may be home at the number selected, or the selected individual within the household may not be home. In this day and age, for a typical market research or polling survey conducted by phone, one has to load 30 to 40 times as much sample into the system as interviews required. So what happened between the 1990s and 2014? Are people just busier? Well, maybe, but the more likely explanation is more complicated. First, there is a wider proliferation of caller ID. Some folks just will not answer when they see it is a survey research firm calling. In addition, there are fewer folks with landline telephones now than 20 years ago. The proliferation of cell phones and their use as a primary telephone creates complications for survey researchers. Since those who purposefully do not answer their phones and those who use a cell phone as their primary communications device are not a random sample of those who are contacted, the resulting interviews are biased and, therefore, are not a true probability sample. For this reason, a margin of error typically should not be calculated.
So, while margins of error are never appropriate to report for panel-based, online samples, they are also almost never appropriate to report for any other type of sample usually reported on in the media or in market research.