Maine Gov. Paul LePage is not backing down from comments he made about Rep. John Lewis boycotting incoming President Donald Trump’s inauguration.

On a radio interview last weekend, LePage said Lewis should remember that Republicans were the advocates for civil rights, the Portland Press Herald reported.

“It was Abraham Lincoln who freed the slaves, it was Rutherford B. Hayes and Ulysses S. Grant who fought against Jim Crow laws,” LePage said. “A simple thank you would suffice.”

On Wednesday, LePage said he would not apologize for the remarks.

“As a matter of fact, I would tell the NAACP they need to apologize to the northern white people who fought and gave their lives to free them,” LePage said.

LePage has a history of making racially inflammatory remarks, such as saying people named “D-Money, Smoothie, Shifty – these types of guys” coming into the state were responsible for the Maine’s drug problem and that they “they impregnate a young white girl before they leave.”

He also previously told the Maine NAACP to “kiss my butt.”