Cache Partition wipes can take 5min or more on the Nexus 5.
I’ve never known it take longer than a few seconds on any Android device to clear the app cache, whether one, or all at once.
Some say it’s good to clear the app cache weekly, and the system cache monthly; others say no need unless problems arise.

I know that under KitKat and Lollipop it would take 10-20 minutes to a wipe of the cache partition from recovery. I thought that was "fixed" under Marshmallow to be done faster, but I may be thinking of the Turbo 2?

Unless you have problems that seem to be fixed with a wipe of the cache partition, I'd say that once a month is a reasonable interval, and, if you miss a month, it won't be that big a deal.