Caterpillars leave tell-tale droppings and the most common caterpillar pest of tomatoes is the tomato hornworm. These are the larvae of the Sphinx moth and they can get quite large - about 3" in length. They are the same color green as the leaves and stems on tomato plants so they can be munching away and even when very large, you may not see them. A hungry hornworm can defoliate a tomato plant in a matter of hours. Go out and check your plants, looking along stems and under leaves. If you find the caterpillars, hand pick them and get them out of the garden. The botanical Bt (Bacillus thuringensis sold as Dipel) will kill the pests. Apply according to label directions.