This restaurant is right across the street from the south entrance. They have a taqueria next door (less $) that is set up for takeout, but they have 6 or so tables inside and a picnic table outside. You could even take your food up to the High Line. They have a nice shaded spot not too far in with tables.

I think you're talking about the new High Line walk, right? Spotted Pig is close to the southern entrance and open for lunch, but I consider it to be on the pricier side.

Cookshop (near the northern, not southern, entrance) may be an option for you. Lunch fare is not inexpensive, but there are reasonably priced options (i.e. the smoked Atlantic salmon with salad, cream cheese and bialy plate), portions are large and ingredients are high quality. They had outdoor seating as of last week.

Chelsea Market is also right at the northern entrance and offers many inexpensive takeout options.

And on the High Line walk itself (somewhere in the middle, I think), there's a 'wichcraft stand and a nice looking cupcake stand that uses only organic diary in the cupcakes. There are a few seats and tables nearby.

Yes, there are definitely little cafe tables and chairs for eating near the 16th street entrance of the high line (only about 4 blocks from the bottom, making it right in the middle of the part of the high line that's open) so you can grab a picnic and chelsea market, exit the market on the 10th avenue side, and take the elevator right up to the high line, walk just a bit south and you'll see the tables.

If you strictly want to stick to starting at the bottom, there's a Fig & Olive pretty close by that's nice for moms.