Hispanics have overtaken African-Americans as the nation's largest minority. Join columnist Ruben Navarrette and CNN contributor Roland Martin on CNN.com Live Video at 12:30 p.m. ET Wednesday when they will discuss this issue and some of your "Sound Off" comments.

SAN DIEGO, California (CNN) -- Welcome to the black-brown thing. That's what my African-American friends and I called it back in college. It's shorthand for the uneasy relationship between the nation's largest minority and the group that formerly held the title.

Ruben Navarrette Jr.: It's dumb for African-Americans and Latinos to fight with each other.

In college, Latinos and African-Americans got along well. Ours was a small and prestigious school in the Northeast where neither group was in large supply.

As I walked through campus, if I passed a black classmate -- even one I didn't know -- he'd acknowledge me with a smile and a nod. And I'd do the same.

These days, the groups trade elbows. It's wasted energy. I like a good fight as much as the next columnist, but I won't fight for crumbs with people who -- like my own -- had to scratch and claw for everything they have, only to be told that they don't deserve it.

My first experience with the friction between blacks and browns came in 1994. I was co-hosting a daily talk show for ABC Radio in Los Angeles with Tavis Smiley, an African-American who is now a nationally known media personality and the host of a nightly talk show on PBS.

Since we were both in our 20s, we were supposed to talk about the issues from a young perspective. But this being Los Angeles, and one of the issues being immigration, the show disintegrated into black-brown conflict.

Don't Miss

I heard plenty of racist comments -- and most of them came from self-identified black callers and were aimed at Latinos. It's not that Latinos can't be racist toward blacks. They can be. But, with Latinos under siege in the immigration debate, maybe they were too busy defending themselves to take shots at African-Americans.

In 2001, while I was writing for the Dallas Morning News, a videotape surfaced showing African-American City Council candidate Dwayne Caraway delivering a warning to black business leaders: "You better wake up and look at your next-door neighbor," Caraway told the audience. "Because now, your next-door neighbor is Hispanic. And they're moving in. And they're taking over. And they're pushing us out."

When Caraway did well enough in the primary election to earn a spot in a runoff (which he lost), I wrote a column lamenting that the candidate found a market for his fear mongering and noting that the same was true for George Wallace.

Today, one of the battlegrounds is New Orleans, where -- in October 2005, or two months after Hurricane Katrina -- Mayor Ray Nagin complained to a business group that his once predominantly black city was being "overrun by Mexican workers."

Could that have had anything to do with the fact that, for months after Katrina, many African-Americans weren't interested in moving back to New Orleans, let alone in doing the tough, dirty and low-paying jobs that Latino immigrants did to rebuild the city?

Ah yes, immigration. African-Americans are frustrated because they think they're losing jobs to illegal immigrants. Latinos are frustrated that African-Americans -- of all people -- can't detect the racism that poisons this debate.

We got a taste of that recently when black radio talk-show host Warren Ballentine, appearing on CNN's "Lou Dobbs Tonight," warned that if we didn't stop illegal immigration, America's children would one day "be speaking Spanish."

Latinos make the mistake of not paying respect to the unique place that African-Americans occupy in this country and its history. African-Americans make the mistake of assuming that Latinos want to take their slice of pie, when Latinos just want a slice of their own.

It's dumb to think of the American consciousness as a finite commodity, where -- in order for one group to earn recognition -- another has to head to the back of the line.

That's pretty much what another black radio talk-show host -- Joe Madison -- said Tuesday on CNN's "Lou Dobbs Tonight" when he accused Democrats of being so eager to court Latino voters that they've banished blacks to a "lower tier."

Please. Or, as they say in the new America, por favor.

Ruben Navarrette Jr. is a member of the editorial board of The San Diego Union-Tribune and a nationally syndicated columnist.You can read his column here.

The opinions expressed in this commentary are solely those of the writer.E-mail to a friend