The Public Editor: When the subject is a poll's accuracy, size matters

The competitive and volatile campaign for the presidency, which brings its traveling primary show to California in nine days, has focused renewed attention on voter polls.

As in how accurate are they?

They certainly took a well-publicized and richly deserved hit earlier this month when they heralded Sen. Barack Obama's impending victory in New Hampshire's Democratic primary, only to be blind-sided and embarrassed by Sen. Hillary Clinton's surprising resurrection.

Surprising at least to the pollsters and pundits.

The political landscape is still shaking from that debacle, aftershocks felt right here in River City.

Last Tuesday and Wednesday, The Bee published on the front page the latest California Field Poll handicapping the Democratic and Republican primaries.

The first story said Clinton was maintaining her double-digit lead over Obama. The second story put Republican Sen. John McCain in front by nose over Mitt Romney, with a margin of error of 5.2 percentage points.

You had to get to the fine print buried in the accompanying charts to find out that each poll was based on the responses of only 377 likely voters.

Wow, that seems skimpy to me, and the sort of pertinent information that should go into a front-page story. Forcing readers to dig out that information makes it seem like the paper is trying to hide it.

At a minimum, stories should include the size of the poll's sample, the margin of error and when the poll was taken.

Sometimes stories in the paper include that information, but not, coincidentally, when the sample size is very small, as it was last week.

But it wasn't the smallest sampling size used recently. On Dec. 20, The Bee printed a front-page Field Poll story about GOP presidential candidates based on a survey of 322 people, with a margin of error of 5.7 percentage points.

Those details weren't in the story.

The result is a few readers, such as retired geologist Robert Sydnor of Fair Oaks, are wondering how such small samples could accurately represent the opinions of all California voters.

"To me it seems alarmingly small," Sydnor said after seeing the 377 respondents in the Democratic poll. How tiny, he asked, would the sample have to be before The Bee refused to publish the poll?

It appears to me it might be single digits.

Given the state has 36 million people, with 15.5 million registered voters, almost 9 million of whom cast ballots in the last general election, Sydnor's raises a legitimate question.

Mark DiCamillo is the director of the Field Poll, a nonpartisan survey of California public opinion that began in 1947. He's been the director since 1992.

He said he'd like to have larger sample sizes too, but he's constrained by the primary and his budget.

Because the primary is split between Democrats and Republicans, he said, the poll has to survey twice.

To keep the sample sizes consistent, that also requires over-sampling Republicans, who are a smaller percentage of registered voters than Democrats.

Poll workers interviewed more than 1,000 registered voters before whittling them down to the survey's 377 likely voters, with a margin of error of 5.2 percentage points, DiCamillo said.

Respondents were asked questions to evaluate how likely they were to vote, such as whether they asked for or received absentee ballots. Twelve percent of those in the sample said they had already mailed in their ballots.

And then there's the budget. The Field Poll is funded in part by many of the state's newspapers, including The Bee, which are charged based on circulation size. It is a long-standing partnership.

DiCamillo declined to say how much each paper pays, saying the poll's overall budget this year is more than $300,000. The papers' slice of the budget has diminished from 20 years ago.

With a plan to do five statewide polls this year, it falls on DiCamillo to make the budget work. And so there are trade-offs: like doing polls with smaller sample sizes and higher margins of error. Where the effect is most noticeable, he said, is in subgroup analysis, such as comparing demographics based on age and regional differences.

What the Field Poll is intended to show, he explained, are trend lines, not final election percentages. The poll shows direction of change over time and which candidates have momentum. The results are grist for further analysis about what they mean.

Dan Smith is The Bee's state Capitol bureau chief. His reporters write the Field Poll stories.

As one of the papers paying for the poll, The Bee has a voice in selecting topics. The poll provides respondents who have agreed to talk to reporters.

Smith also would like larger sample sizes, preferably of at least 1,000 people, which would give the poll a smaller margin of error and make it more reliable.

He agreed the paper's been inconsistent in noting sample sizes in its stories.

The working rule for a long time, Smith said, has been that mention of the sample size, interview dates and margin of error should be confined to the charts accompanying a poll story.

Part of the reason, he said, is that it's considered repetitive to have the information in both places. "But," Smith acknowledged, "we shouldn't have a hard and fast rule."

Given the paper is already inconsistent – sometimes mentioning sample size and margins of error and sometimes not – I think it would be better for poll stories to include such pertinent information.

That let's readers decide for themselves how seriously to consider the poll and the story.

So what if it's repetitive – remember, we're talking fine print as it stands now – if it helps readers better understand what they're being told?

What do you think? Send me your comments and questions for posting online at the Public Editor's forum.

To participate, please go to www.sacbee.com/public.

About the writer:

• The Public Editor deals with complaints and concerns about The Sacramento Bee's content. His opinions are his own. You can contact the Public Editor by mail at P.O. Box 15779, Sacramento, CA 95852; or by calling him directly at (916) 321-1250.