House Budget Chairman Paul Ryan acknowledged Sunday that his budget plan has drawn sharp reactions from constituents back home in Wisconsin, which the Republican said is “a sign of the times.”

“The crowds are really getting bigger” at the town halls he has been holding over Congress’ two-week recess, Ryan said during an extended interview on ABC’s “This Week.” “And people are getting much more anxious about just where the country’s headed.”

Ryan, whose far-reaching budget plan was approved by the House along party lines last month, also shrugged off suggestions of defections from his own party. Specifically, he said he “didn’t take it personally” when Speaker John Boehner (R-Ohio) said last week that he was “not wedded” to the Ryan budget.

“No, not at all. I didn’t take it personally,” Ryan said when asked whether he was offended by Boehner’s comments. “It’s not — it wasn’t meant to be personal. I don’t take it that way.”

He maintained that his plan, which would privatize Medicare, aims to help retirees, and he decried the “TV, radio and phone calls that are running, trying to scare seniors.”

Democrats have hammered the GOP on Ryan’s plan and used it as a messaging tool over the course of the spring recess. The budget proposal has sparked fireworks at town hall meetings throughout the country, and Ryan was even heckled by constituents who charged that his plan was too drastic.

Ryan, a rising star in the Republican Party, acknowledged that his plan is politically risky but that the issue can’t be ignored.

“I hear this all the time from the political people, from the pundits and the pollsters that this could be — this could hurt us politically,” Ryan said. “I don’t care about that. What I care about is fixing this country and getting this debt situation under control. … If all we fear about is our political careers, then we have no business having these jobs. If you want to be good at these jobs, you’ve got to be willing to lose the job.”