Even though you figured it out, this is for anyone else with the same question.

Your palace generates laborers. Each level of your palace, as it is upgraded, generates a new batch of laborers. So the trick is to keep upgrading your palace to get more laborers.

Of course, you have to feed any workers that have actual jobs, so you will have to keep your bakery and wheat fields upgraded to keep ahead of the problem.

I used to like to keep my wheatfields and bakery upgraded to two levels higher than the palace while building my city. That way I could harvest plenty of wheat and make lots of bread to feed the workers and maybe have some extra for purchases.

If you send workers to a monument site from one of your cities, the amount you send will be re-generated by the palace so you will always have the maximum number of laborers your current palace level allows.

Any labor you send to the monument site (including soldiers) have to be fed so you have to keep sending enough bread to keep any laborers there engaged in actual work fed.

You can leave workers unemployed and you won't have to feed them. If you don't feed the workers and/or soldiers at the monument site, they will abandon the site and leave. You'll have to send more workers/soldiers there to replace the one's you lost, if you still want to keep the site producing work on a monument or a quarry or have soldiers there to protect the site.

Ultimately, your palace will top out at level 22 and you'll have all the laborers that you are ever going to get so use them wisely.