North Carolina Rep. Robert Pittenger has filed a lawsuit against a would-be neighbor over the size of the home he’s building next door to Pittenger’s.

The outgoing congressman, who lost in his re-election primary last month,  and his wife are suing Doug Lebda, CEO of Lending Tree, who is building a home near the Quail Hollow golf course, WSOC TV reported.

But Pittenger and his wife say in their lawsuit that the homeowner’s association “knowingly approved” the home even though they say it violates HOA guidelines.

“The Pittengers, and other owners… will be permanently and irreparably injured if (they) are allowed to construct the planned structures on the property,” the suit said.

The Pittengers are asking the court to rule to “demolish or reconfigure the structures.”

Sources told WSOC that the Pittengers think the Lebdas’ home is two inches short of the 10-foot buffer required between homes and that the HOA gave the Lebdas “unintended overage variance.”

“Punitive damages are appropriate in this case to punish Defendants… for their egregiously wrongful acts,” the lawsuit said.

A source familiar with the Lebdas said they redesigned a roof overhang and changed windows and even offered to buy 50 square feet from the Pittengers to settle the dispute.

WSOC reported that documents it reviewed showed the Pittengers were asking for $1.725 million for the 50 square feet of property.

“The Pittengers are entitled to monetary damages,” the lawsuit said and cites “emotional stress suffered” as a reason.

Jamie Bowers, Pittenger’s spokesperson said the matter was not a congressional issue.

Pittenger in May lost the Republican primary for North Carolina’s 9th District to Pastor Mark Harris.

Watch: Seersucker, Hemp and the Caps: Congressional Hits and Misses