Rep. Heath Shuler plans to vote for himself rather than Rep. Nancy Pelosi during Wednesday’s floor vote for Speaker, and he predicted Monday that Democratic colleagues who are similarly displeased with her past leadership as Speaker would back him as well.

Shuler and some other House Democrats blame Pelosi for the drubbing their party took in the midterm elections, and they accuse the Californian of not being responsive to moderate Democrats, who suffered significant losses in the election. “I personally feel and I think others still feel like, even after what happened in the last election, we’re still not being heard,” the North Carolina Democrat said Monday. “The American people heard it. That’s for sure. … We think we represent the masses.”

Defections from Pelosi to another Democrat would be symbolic only: Republicans have the votes to elect their designate, John Boehner (Ohio), as Speaker by a wide margin. But Shuler and his allies have argued that it is important to send a message to the leadership.

“I think it’s significant,” said a senior Democratic aide with ties to moderate members. “We all need to move forward, but that doesn’t mean that people have forgotten how many of their colleagues aren’t here with them.”

Shuler managed to win the support of 43 colleagues when he challenged Pelosi in November for the title of Minority Leader in the 112th Congress, which begins Wednesday. He said Monday that he was “aware of others who are going to be casting their vote” for him rather than Pelosi on Wednesday, although the former NFL quarterback declined to predict how many he would get.

Although Shuler said he was not whipping his colleagues, he has already garnered the public support of several fellow leaders of the fiscally conservative Blue Dog Coalition, including Reps. Jim Matheson (D-Utah) and Mike Ross (D-Ark.). Rep. Steny Hoyer (D-Md.) recently tapped Matheson to join his Minority Whip team as a chief deputy whip.

The senior aide and a Democratic leadership aide both predicted Monday that a chunk of Democrats will register their discontent with Pelosi’s leadership as Speaker during the past four years by publicly backing Shuler or another Democrat.

“I don’t think it will be 40, but I wouldn’t be surprised if it were 20,” the senior aide said. Because the floor vote for Speaker is public — unlike the Caucus vote, which is by secret ballot — the stakes would be much higher for a Democrat to vote for someone other than Pelosi, the aide noted.

The vote for Speaker at the start of each Congress is conducted as a live roll call, during which Members are called upon in alphabetical order to state whom they want as Speaker. Members can vote for anyone colleague, but traditionally the top Republican and Democrat are nominated and garner the support of most, if not all, of their party’s members. Rep. John Dingell (Mich.), the longest-serving House Member, might be among the Democrats who draw votes on the floor from those hesitant to back Pelosi, the aide said.

The leadership aide predicted that Pelosi would not try to discourage Members from backing another Democrat.

“I would be surprised if there was any whip operation,” the aide said. “Pelosi needs to pick her battles. This vote doesn’t really matter. She will probably just let it go and downplay it.”

Anna Palmer contributed to this report.