Elections & Democracy

Polling: How It Works and Why It Misses

1936— present

Every poll is an estimate about the people who did not answer.

A poll is not a measurement of public opinion. It is a small sample of the people who agreed to answer, adjusted by hand until the group resembles the country — and both of those steps are where the error lives. In 1936 a magazine mailed ten million ballots, got 2.4 million back, and called the election for the wrong man by 24 points; George Gallup beat it with a sample a fiftieth the size and then missed 1948 badly enough that the profession commissioned a 400-page inquest into itself. By 2024 the machinery was unrecognizable and the failure mode was not: the average national presidential poll missed the margin by 2.6 points — ordinary by historical standards — and still overstated the Democrat, for the third presidential election running. This explains how a poll is actually built, what the margin of error does and does not cover, the full record of error from 1936 to 2024, why telephone response rates fell from 36% to 6% without necessarily making polls worse, and the honest disagreement over whether the last three cycles are a broken instrument or a run of the same luck.

Gallup's miss on the 1948 margin

9.4 pts

Its final poll had Dewey ahead by 5; Truman won by 4.4. The inquest that followed rebuilt the profession.

2024 national presidential polls

+2.5 pts

Average signed error — polls overstated the Democratic margin, the third presidential cycle running (AAPOR, 60 polls)

Pew telephone response rate

36% → 6%

1997 to 2018, on the AAPOR RR3 definition. Pew then moved its U.S. polling online and stopped publishing the series.

2024 state presidential polls

3.0 pts

Average absolute error on the margin — the smallest for any presidential cycle since 1944 (AAPOR, 291 polls)

Methods that reliably reduced error

none found

AAPOR tested sampling frame, mode, weighting variables and likely-voter models across 611 polls and found no recipe that guaranteed accuracy

What a poll actually is

A poll does not measure what the country thinks. It measures what a few hundred or a few thousand people who agreed to answer said, and then adjusts that answer by hand until the group resembles the country on the characteristics the pollster chose to correct for. Both halves of that sentence are where the error comes from. The published margin of error describes neither of them: it covers only the luck of the draw in picking names out of a list, and it assumes every other step went perfectly.

STEP 1The frame

The list the sample is drawn from — phone numbers, addresses, a voter file, or the members of a standing panel. Nobody outside the frame can be sampled at all.

HOW IT FAILS

Coverage error. If the list systematically excludes a kind of person, no amount of sampling fixes it.

A DOCUMENTED CASE

The Literary Digest mailed ten million ballots in 1936 to names taken from telephone books, club rosters and registration lists — a frame skewed toward people who had a telephone in the seventh year of the Depression.

STEP 2The sample

Who gets picked out of the frame. Under probability sampling every person has a known chance of selection, which is what makes a margin of error meaningful.

HOW IT FAILS

Sampling error — the ordinary luck of the draw. This is the ONLY error the margin of error describes, and it shrinks predictably as the sample grows.

A DOCUMENTED CASE

Gallup beat the Digest in 1936 with roughly fifty thousand interviews against its 2.4 million returns. A bigger sample of the wrong people is still the wrong people.

STEP 3The contact

Reaching the selected person and persuading them to answer. Measured as the response rate: the share of sampled households that yield an interview.

HOW IT FAILS

Nonresponse error. It only becomes bias when willingness to answer is related to the thing being measured.

A DOCUMENTED CASE

Pew's typical telephone response rate fell from 36% in 1997 to 6% in 2018. Across that span its estimates of party identification and religious affiliation kept tracking high-response in-person surveys — while its estimates of volunteering did not.

STEP 4The weights

Adjusting the people who did answer so the sample matches known population totals — age, sex, race, region, education, and increasingly party and past vote.

HOW IT FAILS

You can only correct for a variable you weight on, and only if you know its true population value.

A DOCUMENTED CASE

The 2016 state polls that missed were much less likely than national polls to weight on education, at a moment when education had become one of the strongest predictors of vote choice.

STEP 5The likely-voter model

Deciding which respondents will actually vote, by self-report, by past turnout on a voter file, or by a multivariate model.

HOW IT FAILS

It is a forecast of behavior that has not happened yet, which is the one thing a survey cannot observe.

A DOCUMENTED CASE

In 2024 counties that backed Trump in 2020 turned out at higher rates than polls assumed and Biden-leaning counties lower. AAPOR found that turnout mis-projection explains a moderate share of the remaining directional error.

The margin of error is the smallest of the errors

A poll of 1,000 people is usually reported with a margin of error around plus or minus 3 points. That figure is arithmetic: it is what the spread of results would be if you drew the same random sample from the same list over and over. It says nothing about a list that leaves people out, about the difference between the people who answered and the people who did not, about weighting to the wrong target, or about a wrong guess as to who will turn out. Those four are together called total survey error, and they are what actually went wrong in 2016, in 2020 and in 2024. A poll can be within its margin of error and still be wrong by twice it.

Two further consequences worth stating plainly. First, the margin of error applies to the whole sample: a crosstab of 180 Hispanic respondents inside a 1,000-person poll carries a far wider one, which is why AAPOR found pollsters who agreed on the topline in 2024 disagreeing by as much as 20 points on the Hispanic vote margin. Second, on the margin between two candidates the uncertainty is roughly double the headline figure, because both candidates’ shares can move.

The response rate you are shown, and the one you are not

For a telephone poll the response rate is the share of sampled households that produced an interview, and it is computable. For an online opt-in panel it is not: far more people are invited to join a panel than ever do, and nobody knows the denominator. What such polls publish is a completion or participation rate, which is a different quantity, and Pew notes this makes them not comparable to telephone polls on that metric at all. Even a probability-based online panel splits the number two ways — one Pew American Trends Panel wave reported a 94% response rate for that wave and a 5% cumulative rate once non-response to recruitment and attrition were counted. Both are correctly calculated. Only one of them tends to get quoted.