In all, 51 percent of Americans now express explicit anti-black attitudes, compared with 48 percent in a similar 2008 survey.

When measured by an implicit racial attitudes test, the number of Americans with anti-black sentiments jumped to 56 percent, up from 49 percent during the last presidential election. In both tests, the share of Americans expressing pro-black attitudes fell.

I knew, even as I read the poll and watched some of the alarmist reactions to it, that such fears were overwrought. It’s hard for me to accept that white Americans en masse really have grown more antagonistic toward black Americans over the past four years. What’s the evidence? One poll, which seemed to contradict common sense and negate all that we see around us? The handwringing over the AP poll brought to mind the punch line of an old Richard Pryor joke: Who are you going to believe? The AP? Or your lying eyes?

The overwhelming majority of what I see around me suggests that we’re moving in the right direction on race. There’s a lot of talk about race nowadays. Some of it’s bad, especially if you’re prone to listening to right-wing radio. But even that’s a sign of progress, I’d argue. In the bad old days, nobody talked openly about racial and ethnic differences. The absence of such conversations wasn’t harmony. No, it was based on a fallacious assumption that only white people mattered in a nation that has always been a polyglot of humanity.

No one can make such a mistake now. Not after 2008, when for the first time in our nation’s history a black man was freely and fairly elected as the national leader. It’s an even more ridiculous claim in 2012, when President Obama repeated that feat with a resounding re-election.

Indeed, there should be more—not less—talk and debate over racial and ethnic distinctions. The increase in conversations is a sign of a healthy democracy, not a weakness in our political system. Sure, the conversations may get rough and ugly at times, but that’s part of the process of learning to embrace differences without having to take it out back and use violence to settle the score.

Those who doubt that racism is waning, not waxing, need only look around with wide eyes and open minds. Along with the constant image of a black president in the White House, the nation and world are reminded of how far the United States has come with daily images of a dark-skinned First Lady and black children growing up in front of omnipresent cameras. In a nation that once enslaved such people, and only during many of our lifetimes allowed them to vote, this has to say something about the distance that white American attitudes have traveled to embrace this reality.

It can only get better. For generations to come, young people—black and white alike—will see less and less what is special or novel about the color of the First Family’s skin. It may be difficult for some today to comprehend, but as time marches on and old people die with their racist blinders affixed, the nation’s racial hang-ups will fade.

If I may play devil’s advocate, though, there’s an argument that having a black family in the White House may provoke those sad individuals who cling to racist sentiments, compelling them to be all the more outspoken. Perhaps, but I don’t buy it. Anyone bold enough to publicly exclaim overt racism is more likely to be shouted down and ostracized by a greater number of Americans than the tiny few who will cheer them on. In generations past that simply wasn’t the case, as open hostility was the norm, especially for Southern politicians. What passes for racism nowadays is hidden and shrouded with alternative explanations; nobody wants to be labeled a racist.

Let me be clear and offer an obligatory disclaimer: Racism does exist. And, sadly, I’ll allow that it will continue in some form for as long as people are, well, human. But thus far into the 21st century, the problem swirling about race talk and race feelings is predominately one of unrealized expectations for our better selves. We want race to go away and never bother us again. Still, only a fool would deny that we’re better on the racism front today than we were 25, 50, or 100 years ago.

So why would anyone believe that the past four years, with President Obama in the White House, mark some sort of reversal of an ongoing positive trend?

Perhaps it’s because we’re not moving fast, far, and frenetically enough to outpace a frustrated nation’s desire to erase America’s original sin. Maybe our history of improving race relations needs a press agent.

But it begs the question: Why do so many of us pay attention to the noise that contradicts the reality that surrounds us? I call it the “Freddy Krueger Theory” of race relations: Simply put, some folks like to be frightened by strong (read: racial) fears. They enjoy lamenting the changing state of “their” country, shuddering at the idea that it’s being overtaken and permanently changed by “others.” Remember Freddy? He was the serial killer in the series of “Nightmare on Elm Street” movies, who killed people in their dreams and thereby caused their deaths in the real world.

The same theory applies with a racial boogeyman. This dream-like fear mongering is effective in politics—in all directions. If we believe that racism is the sub rosa explanation for most of American behavior, then it’s easy to be frightened by racial differences. I think that’s what’s going on here, and an easy-to-believe poll helps makes the story all the more frightening.

In other words, some of us just like to be scared witless over things that just aren’t real.

Sam Fulwood III is a Senior Fellow at the Center for American Progress and Director of the CAP Leadership Institute. His work with the Center’s Progress 2050 project examines the impact of policies on the nation when there will be no clear racial or ethnic majority by the year 2050.