servants are not needed, but help a lot. Noble and educated workers' houses with servants are happier since they can shop larger quantities of goods, luxury shopkeepers use servants to collect raw materials.
Moreover are the first to upgrade when you place a better job, like shop keeper, so you don't vacate farms. Last they help population growth.
I usually start with at least 4 servants.