This paper presents a hardware architecture for increased performance of color classification. In our architecture, color classification, based on an AdaBoost algorithm, identifies a pixel as having the color of interest or not. We designed the proposed architecture using Verilog HDL and implemented the design in a Xilinx Virtex-5 FPGA. The architecture for color classification can have 598 times performance improvement over an equivalent software solution and 1.9 times performance improvement over the leading hardware color classifier.