Updated July 25 at 2:26 PM |Bickering over polls is a time-honored tradition in campaigns. But the latest exchange of surveys in Utah’s 4th District is not just an example of partisan pollsters getting very different results. It is also a reminder of the importance of polling methodology.
Earlier this month, the National Republican Congressional Committee released an interactive voice response survey, conducted May 15-16 by a conservative outlet, Harper Polling. It showed Rep. Jim Matheson, D-Utah, leading his GOP challenger, Mia Love, 44 percent to 41 percent.
Some Democrats grumbled about the validity of the survey.
According to a write-up in The Salt Lake Tribune, the general-election ballot test came after a question that asked if respondents wanted "a Republican who will be a check and balance to President Obama or a Democratic candidate who will help President Obama to pass his agenda?" Unsurprisingly, Utahns chose the "check and balance" candidate, 56 percent to 27 percent.
Democratic strategists believed the question order tainted the purity of the ballot test question. They also countered with a survey of their own.
The Democratic Congressional Campaign Committee conducted its own IVR poll (June 4-5), which showed Matheson with a sizable, 54 percent to 40 percent, lead over Love.
Seven months ago, Matheson defeated Love by 768 votes, 48.8 percent to 48.5 percent.
In the DCCC survey, the initial ballot test was the third question of the poll. Respondents were first asked whether they had a favorable or unfavorable view of Matheson and Love and then who they would vote for if the election “were held today.”
Multiple veteran pollsters on both sides agree that if the check-and-balance question was asked first, it would likely “taint” the ballot test between the candidates.
But according to Brock McCleary, founder of Harper Polling and former NRCC polling director, his survey started with a generic ballot, then the initial head-to-head, and then the check-and-balance question. That order is different from the order of the questions in the memo (generic, check-and-balance, ballot test) that was released to selected reporters and likely resulted in the confusion.
The end result of this squabble are two partisan polls with two very different views of a race. No breaking news there.
But why does question order matter?
“Any poli-sci class on polling will teach the order principle of moving from general to specific as the more specific a question the greater chance that it will bias what follows,” one GOP pollster said. “That said, you have to balance that with the need to protect the integrity of the ballot test.”
The check-and-balance question is more broadly referred to as a “motive ballot.” Because it “informs” the respondents by giving them information that they would not have had on their own, it could taint the ballot test if it precedes it, according to multiple GOP pollsters.
According to pollsters, the motive ballot often improves partisan intensity and can alter the response of independent voters. In this specific case, injecting President Barack Obama in a district where he received 30 percent of the vote had the potential to influence the ballot test.
In contrast, testing name identification and favorable/unfavorable ratings of candidates before the ballot test is common because it is regarded as not imparting any new information, and the name in a name ID question is often the same as the name voters would see on a ballot.
The motive ballot is certainly not unique to the Harper survey, but it is not a basic metric, such as ballot strength, job approval rating and name identification. The motive ballot can be instructive for targeting, and campaigns often use a version of the question to help with messaging.
To add another wrinkle to the Utah 4 polling debate, the Center for Study of Elections and Democracy at Brigham Young University released a poll on Tuesday that showed Matheson with very good favorable ratings.
But the poll was of a pre-selected group of voters who were emailed an invitation to participate in an online survey. In short, it wasn’t a random sample and isn’t comparable to other surveys by most professional pollsters.
Utah’s 4th District is currently a Pure Tossup, according to Rothenberg Political Report/Roll Call ratings. For a more extensive analysis of the rematch between Matheson and Love at the Rothenberg Report, click here ($).
Updated July 25 at 2:26 PM | Dr. Quin Monson, director of BYU’s polling center, and one Virginia-based pollster took exception to my characterization that the school’s survey was not a random sample. They maintain that it is. The methodology was linked in the original story and you can see it again here.
I subsequently contacted several veteran, partisan pollsters and asked them to look over the methodology. Some of them shared my concerns about randomness. “[T]heir panel may be random, but that the resulting sample is not,” according to one pollster about the BYU poll, characterizing one consensus critique.
I will leave it to polling experts to ultimately decide what constitutes a random sample. However, the BYU survey is not comparable to a traditional campaign poll.