Rep. Paul Ryan is making poverty a signature issue as he tries to broaden his appeal ahead of a possible presidential run in 2016. But in being more vocal about the issue, he’s had to defend himself and his message against allegations of racism.

At issue are comments the 2012 Republican vice presidential candidate made during an interview last week on a conservative radio talk show in which he said, “We have got this tailspin of culture, in our inner cities in particular, of men not working and just generations of men not even thinking about working or learning to value the culture of work.”

He later called the remarks “inarticulate about the point I was trying to make,” but that hasn’t stopped Democrats from swarming against him. Rep. Barbara Lee, D-Calif., a member of the Congressional Black Caucus, called Ryan’s comment a “thinly veiled racial attack.”

The stir highlights a potential peril for Ryan, now nearly two years removed from a failed national campaign and two years away from a potential 2016 presidential bid.

The Wisconsin congressman had hoped his work on poverty could be a positive: His interest in the issue dates back to his time as a speechwriter working for former vice presidential nominee Jack Kemp.

He has spent much of his time since returning to Congress focused on the issue, touring poor precincts, giving speeches and producing a detailed, 205-page report on poverty, while indicating that he may introduce legislation to deal with the issue.

Instead, Ryan’s remark has brought negative attention to him, highlighted Republicans’ continued struggles to connect with minority voters and stepped on his own policy push.

At home in Wisconsin this week, Ryan sought to clarify his message — that poverty was both a rural and urban issue.

Speaking at a packed town hall in North Prairie, Wis., Ryan warmly thanked a constituent who mentioned the recent controversy and told him she did not think he was racist.

“In our society, we have done a lot to isolate the poor, in rural and urban America,” Ryan said. “This enforces the idea that this is government’s responsibility, and you don’t need to do anything about it. That’s not true.”

Ryan did not mention the rural poor during his appearance on Bill Bennett’s “Morning in America” radio show. Instead, his remarks centered on what he saw as a problem with the culture of inner cities.

“Let’s be clear, when Mr. Ryan says ‘inner city,’ when he says ‘culture’ these are simply code words for what he really means: ‘black,’” said Lee, a member of the House Budget Committee, which Ryan chairs.

Ryan has since called Lee and announced plans to meet with the Congressional Black Caucus after Congress returns from a recess next week.

Any Republican with national aspirations will have to improve the party’s standing with minority voters. In 2012, President Barack Obama won 80 percent of minority voters en route to a comfortable popular vote and electoral college victory while losing white voters.

Ryan hasn’t decided if he will run for president in 2016. But his schedule — which has included trips outside of Wisconsin and speeches like one he gave recently to the Conservative Political Action Conference — indicates he wants a national profile.

Among potential Republican 2016 contenders, Ryan also isn’t alone in facing early struggles. New Jersey Gov. Chris Christie has seen his popularity drop amid an investigation into the closure of a New Jersey bridge for political payback, and Sen. Rand Paul, R-Ky., was accused of plagiarism after a speech he gave bore similarities to a Wikipedia page. Sen. Marco Rubio, R-Fla., has been criticized by conservatives for backing immigration legislation, and Sen. Ted Cruz, R-Texas, has been blamed by some for stoking the unpopular government shutdown.

Republicans defended Ryan this week amid the criticism of his comment to Bennett.

National Republican Chairman Reince Priebus, a fellow Wisconsin native, said Ryan was correct to acknowledge that his comments were inarticulate, but that it was unfair to question his motives. Priebus said people should consider the fact that Ryan was talking about poverty at all.

“Why was Paul talking about this issue?” Priebus said. “He’s talking about this issue because he’s devoted a large part of his life to come up with, I would say, methods, policy, political approaches, all of the above, to help tackle poverty in this country.”

Priebus said Ryan’s efforts on poverty show that Republicans are trying to broaden their appeal and “venturing into areas that our party should venture into.”

But doing so comes at a risk — and Ryan has already proven that, said Mary Berry, a professor at the University of Pennsylvania who served as the chair of the U.S. Commission on Civil Rights from 1993 to 2004.

Republicans must be careful they “don’t alienate their conservative white supporters,” she said. Meanwhile, Democrats are generally happy to pounce on any missteps.

“Democrats will jump all over them in the messaging game, no matter what they say,” Berry said, “and they won’t be given the benefit of the doubt — that’s politics.”