How is work dependent on displacement rather than distance?

Using the definition of work=force x displacement (which I understand is correct):If I push a box along the ground with force of 10N over a distance of 1m and then back again to the starting point, the above eqn would indicate I have done zero work (because displacement =0). But my muscles have put a lot of work into the box (like force x distance)?
I would appreciate it if anyone could explain this?

If you push a box out and then back then this is not a case of constant force. The force times displacement formula only works for a constant force. So break the problem up into two pieces where the force is constant on each piece.

The work you have done pushing the box out is +10N times +1m = 10J. The work you have done pushing the box back is -10N times -1m = 10J. Total work done = 20J.

More generally the work done pushing an object along a path is the "path integral" of the vector dot product of instantaneous force times incremental displacement along the path -- in effect you break the problem down into lots and lots of small pieces where force and direction are approximately constant and then take the limit as "lots and lots" goes to infinity.