Wary of a narrow majority, one of President-elect Joe Biden’s closest allies in Congress isn’t sure the Democratic Caucus can afford to see any more House members join the new administration.

“I think we better bring that to a halt,” Majority Whip James E. Clyburn told reporters Wednesday.

In addition to being the Democratic Party’s top vote-counter in the House, Clyburn is also the chairman of Biden’s Presidential Inaugural Committee, which is in charge of the festivities (both in-person and virtual) away from Capitol Hill.

Clyburn’s endorsement of Biden ahead of his home state South Carolina primary was seen as a key to Biden reviving his campaign.

“Two votes I can handle,” Clyburn said. “I don’t know if I can handle three.”

Louisiana Democratic Rep. Cedric L. Richmond will be leaving the House to become a senior adviser to the president, as well as director of the White House Office of Public Engagement. Rep. Marcia L. Fudge, D-Ohio, is the widely reported choice to be Biden’s nominee to be secretary of Housing and Urban Development.

Richmond’s seat will almost surely be vacated first, since he is in line for a White House position that does not require Senate confirmation. Once both seats are vacated there will potentially be as few 220 members of the Democratic caucus until the seats are filled through special elections.

House Majority Leader Steny H. Hoyer expressed similar concern about the Biden team taking Democrats from the House.

“I hope so,” Hoyer said when asked if he was confident the president-elect had made his last Cabinet pick from the ranks of his caucus.

“Two is too many, but three would would be even more many,” Hoyer said.

The concerns of Hoyer and Clyburn about the math problem could be another bad omen for other House members who could be considered for senior Biden admiration posts, including New Mexico Rep. Deb Haaland, whose name has been floated as a potential Interior secretary pick and would be the first Native American to serve in that capacity.

Clyburn said he did not have any particularly unique concerns about Democratic caucus unity with the narrower majority that waits him in 2021, but he did anticipate that the decision-making about the structure of legislation that reaches the House floor will reflect the reality of the numbers.

“I don’t know if I have any concerns about that, any more than I’ve always had. You know, keeping this party together is just like trying to keep a family together,” he said. “When you’ve got 20 votes to spare, it’s a little different when you’ve only got two.”

Lindsey McPherson contributed to this report.