You're right. Anyone can be a racist. This definition from Dictionary.com explains racism:

"3. hatred or intolerance of another race or other races"

However, remember that in the U.S., historically the whites ruled and racism mostly went against blacks. It was the whites who owned slaves, forbade blacks to eat in white restaurants, attend white schools, and ride in the front of city buses. It was whites who denied jobs to blacks.