The Washington Post-SurveyMonkey 50-state poll was conducted to provide an expansive portrait of American voter attitudes about the 2016 election from every part of the country.
How the survey was conducted
The Post-SurveyMonkey poll used an online-based sampling methodology that differs from previous polls by The Washington Post. Those are telephone surveys based on random samples of cellular and landline phones.
The new poll was conducted online as part of SurveyMonkey’s 2016 Election Tracking project, which recruits respondents from the large number people who take polls on the company’s do-it-yourself survey platform, roughly three million each day. A subsample of respondents to this range of surveys — which includes formal and informal polls of community groups, companies, churches and other organizations — were invited to participate in a second survey with the prompt, “Where do you stand on current events? Share your opinion.” The survey was not advertised on any website, so individuals could not “click-in” in an effort to influence results. A survey invitation could be used only once.
From Aug. 9 to Sept. 1, the survey asked the sample of 74,886 registered voters about their presidential support, including between 546 and 5,147 respondents in each state. The final sample was weighted to the latest Census Bureau benchmarks for the population of registered voters in each state.
The Post-SurveyMonkey poll employed a “non-probability” sample of respondents. While standard Washington Post surveys draw random samples of cellular and landline users to ensure every voter has a chance of being selected, the probability of any given voter being invited to a SurveyMonkey is unknown, and those who do not use the platform do not have a chance of being selected. A margin of sampling error is not calculated for SurveyMonkey results, since this is a statistical property only applicable to randomly sampled surveys.
The ability of random samples to represent the overall population is grounded in probability theory. With non-probability samples, testing is necessary to ensure a particular sampling strategy, along with adjustments to match population demographics, can consistently produce accurate estimates.
Before partnering with SurveyMonkey, The Post sought to assess the accuracy and reliability of the company’s surveys by examining how their previous findings compared with election results, and to high-quality probability sample surveys
This review, detailed below, gave us confidence that its samples are of sufficient quality that they can be useful in shedding light on the opinions of the broader population of voters.
Representativeness of results
The Post has generally avoided citing results from non-probability Internet-based surveys such as SurveyMonkey, as it is impossible to draw a random sample of Internet users, and random selection is a widely accepted standard in drawing representative samples of any population.
As Internet-based surveys have proliferated, research has grown on the ability to make accurate population estimates from these non-probability samples. Several benchmarking studies have found that probability sample surveys produce smaller errors than samples from opt-in, non-probability surveys. But research has also found that some non-probability methods have been more accurate than others. The Post has continuously reviewed this evidence with an aim of developing a standard to determine which non-probability techniques are useful and appropriate.
The first component of our review focused on previous election results. In 2014, the company produced state-level estimates of support for all U.S. Senate and gubernatorial contests using a sample of more than 130,000 adults. Results were not published before the election but were shared and later published by the Huffington Post. Across 71 contests, a Post analysis found the average error in estimates of individual candidate support was 3.2 percentage points, below the five-point average for probability-based live phone surveys. When looking at a different measure of error, the margin between candidates, SurveyMonkey polls differed from the final margin by an average of 4.8 points, slightly lower than 5.6-point average for all live-interviewers surveys.
SurveyMonkey also published pre-election estimates in the 2012 presidential contest. While state-level percentages were not published, the national vote margin of 47.6 percent for Obama vs. 45.7 percent for Romney erred by an average of 2.5 points in estimating each candidate’s actual share of the vote (51.0 percent Obama, vs. 47.2 Romney), and differed from the final vote margin by two points. SurveyMonkey’s estimates heading into the 2016 Super Tuesday primaries erred by four points in candidate support and six points on the margin, though the surveys tended to overstate Trump's and Clinton’s support.
Pre-election poll performance is not necessarily the best measure of accuracy given challenges with likely voter modeling, late shifts in voter attitudes and the potential for polls to gravitate toward other published surveys. As a second level of analysis, The Post reviewed a 2014 study that compared results from SurveyMonkey’s Web panel with results from high response-rate federal surveys. This comparison, conducted by researchers from the Pew Research Center, Westat, the University of Wisconsin and SurveyMonkey, found SurveyMonkey's panel results typically differed from the federal surveys by 6.5 percentage points. That was higher than the 5.4-point difference for a mail survey but lower than the 9.1-point difference for a probability-sample Web survey.
A third gauge of representativeness came from a comparison of SurveyMonkey national results with our own contemporaneous polls. Point estimates between the surveys differed by an average of 5.9 points and a median of 4.4 points. The typical margin of sampling error for the national phone surveys is 3.5 percentage points.
Altogether, our review found SurveyMonkey estimates to be broadly in line with election results, other polling benchmarks and our own trusted cellular and landline phone surveys. These comparisons gave us confidence that results from the sample can be useful in shedding light on the opinions on voters.
The Post-SurveyMonkey survey gave The Post an opportunity to gauge political attitudes on a far greater scale than would have been financially and practically feasible using traditional, probability-based polling. At the same time, The Post remains committed to continuing our long-trusted probability surveys, including those conducted with partners including ABC News, the Kaiser Family Foundation and the University of Maryland. These have a sterling record for methodological rigor and accuracy and will be critical for evaluating the utility of newer methodologies going forward.
