Fact Check: Obama’s 80% of Americans Are 'Sold' Claim Is Skewed

Last week, president Obama made the bold claim that 80 percent of Americans support a resolution that includes a mixture of budget cuts and tax hikes. This seems like an awfully generous number considering that the president's claim is entirely based on a poll. One would assume that he took the poll's margin of error into consideration before he spoke. But this may be hoping for too much.

True, there are several polls that suggest, as Obama said, that Americans are open to a compromise that involves higher taxes as part of the solution. However, claiming that 8 in 10 Americans support it is questionable (to say the least).

"You have 80 percent of the American people who support a balanced approach. Eighty percent of the American people support an approach that includes revenues and includes cuts. So the notion that somehow the American people aren't sold is not the problem,” the president said last week.

The White House has been using a Gallup poll as the main basis for this fantastic claim. However, the Gallup poll tells a different story. According to the original data, 69 percent supported solving the deficit with a mix of spending cuts and tax increases. An additional 4 percent favored tax increases only (this 4 percent is not part of Obama's "balanced" approach and one can safely assume that they are in his pocket regardless of which approach he takes). That brings his support to 73 percent at most in that poll.

But the number 73 only matters if we blindly accept it. One would be foolish to do so. Polls are not definitive by any means and they only give a rough picture. In fact, there are several problems that need to be considered before one accepts the results of any poll. For example, there may be a problem with the sample design (for telephone surveys, how the numbers were selected and how the individuals are selected within the household), non-availability, the refusal problem (is the refusal rate different on the particular variable we are measuring?), question wording, question order, deliberate, or unconscious, lying or false reporting by respondents, or inappropriate or inadequate weighing of data.

All of these variables have been shown in various studies to be the source of not just small errors but sometimes quite substantial ones. Unfortunately, there are not many methods that allow one to quantify the effects of these errors or to validate the results within any kind of reasonable measure. Out of everything that can go wrong in a survey, only the “sampling error” can be quantified

So how did Gallup go about surveying Americans to find out where they stood on the budget crisis? According to Gallup’s website:

“Results are based on telephone interviews conducted April 7-11, 2011 with a random sample of –1,077—adults, aged 18+, living in all 50 U.S. states and the District of Columbia.

Interviews are conducted with respondents on landline telephones and cellular phones, with interviews conducted in Spanish for respondents who are primarily Spanish-speaking. Each sample includes a minimum quota of 400 cell phone respondents and 600 landline respondents, with additional minimum quotas among landline respondents for gender within region. Landline numbers are chosen at random among listed telephone numbers, cell phone numbers are selected using random-digit dial methods. Landline respondents are chosen at random within each household on the basis of which member had the most recent birthday.

Samples are weighted by gender, age, race, Hispanic ethnicity, education, region, adults in the household, and phone status (cell phone only/landline only/both, having an unlisted landline number, and being cell phone mostly). Demographic weighting targets are based on the March 2010 Current Population Survey figures for the age 18+ non-institutionalized population living in U.S. telephone households. All reported margins of sampling error include the computed design effects for weighting and sample design.

In addition to sampling error, question wording and practical difficulties in conducting surveys can introduce error or bias into the findings of public opinion polls [emphasis added].”

The Gallup report itself says that there are possible errors and bias in its findings. However, it does not mention issues that may skew results such as refusal problems, non-availability issues, deliberate, or unconscious, lying or false reporting by respondents, or inappropriate or inadequate weighting of the data, coverage error, measurement error, and non-response error.

Furthermore, the report's “margin of error” presents another problem. Most surveys report margin of error thus: "The results of this survey are accurate at the 95% confidence level - or + 3 percentage points." This error is the statistical imprecision that results from interviewing a random sample instead of the entire population.

The Gallup report only emphasizes sampling error, with a margin of error of plus or minus 4 percentage points, and does little to address the wide range of other opportunities for something to go wrong.

What we know that the "80 percent" claim is false. What is yet to be determined is just how off the mark it really is.

With all of these possible errors, wouldn’t one expect the president of the United States to make sure that he had an iron-clad case for his "balanced approach"before he spoke? One wouldn't expect him to misquote a poll that may suffer from bias and statistical errors. Furthermore, one wouldn't expect him to base fiscal policies with long-term repercussions on that same poll. Perhaps the leader of a representative government should not rely on data pulled from random samplings but should instead make decisions based on the overall population. Crazy thought: we could take a vote.

If the president were honestly interested in knowing what Americans thought about including tax hikes into the budget conversation, then perhaps he could jog his memory and recall the "shellacking" of November. Thousands of voters voiced their opinion in no uncertain terms.

The president has been associated with the word “audacity” on numerous occasions. Maybe this is why.