Don't Panic! You don't need to run out to Office Depot to get a bunch of crazy stickers for your household chemicals--"Each person offering a motor carrier a hazardous material for transportation by highway shall provide to the motor carrier the required placards for the material being offered prior to or at the same time the material is offered for transportation, unless the carrier's motor vehicle is already placarded for the material as required by this subpart." So you do have to have placards, but you can get them from the same place you get your hazardous materials.

But which materials need to be placarded? I'm so glad you asked. 49 CFR 172 (subpart F) defines the following classes of hazmat placards and the regulations that apply:

Class 1: Explosives - placard must be orange with black lettering, bearing the appropriate division number (1,2,3...6) and the word "EXPLOSIVES".

A material, liquid or solid, that causes visible destruction or
irreversible alteration to human skin or a liquid that has a severe
corrosion rate on steel or aluminum.

Class 9: Miscellaneous - white with seven vertical black stripes coverin the upper half. Number "9" is underlined and centered at the bottom.

A material which presents a hazard during transport, but which
is not included in any other hazard class (such as a hazardous substance
or a hazardous waste).

"Dangerous": white placard with upper and lower third in red, with black lettering "DANGEROUS" through the white center. Used when 2 or more from different categories are present in the same shipment (e.g. if 1.x and 2.x are together, but not necessary if 1.x and 1.y are together).

Also:

Placards must be square, and measure at least 273mm on each edge, with a solid line inner border at least 12.7mm from the edge. For other than Class 7 or the DANGEROUS placard, text indicating a hazard (for example, "FLAMMABLE") is not required. Text may be omitted from the OXYGEN placard only if the specific identification number is displayed on the placard.