Will it help your users perception performance? Will that depends. If CPU contention is high (CPU ready), or Memory overcommit is causing an impact (nonzero ballooning or paging). Then there will be a visible and marked improvement.

If not, then there will be an improvement that only benchmarks will pick up, and you need to be looking elsewhere.

Even supposedly "idle" machines run occasional background tasks that consume CPU, memory, disk, network, etc. If it doesn't need to be running, and you're as resource-constrained as you say you are, shut it off.

It will probably help, yes, especially if you're over-committed, especially if you're over-committed on CPU and running multi-CPU guests.

Contrary to popular opinion, adding more vCPUs to a VMWare guest generally degrades performance instead of improving it. Either all the vCPUs assigned are able to run, or none are, so adding more vCPUs to a VM makes it less likely that the VM will be able to access enough physical CPU cores at any given time. Essentially, CPU requests from a guest OS wait in line until there as many physical cores free as the VM has vCPUs.

So, if you've been adding more cores to try to "improve performance," you've actually been increasing queue times for CPU resources and making performance in your guest OSes worse. Assuming that's the case, your first step should be a quick check of your CPU contention and queue times, followed by switching everything that doesn't absolutely need multiple vCPUs over to a single vCPU OS. Depending on the particular OS in qusetion, this may be best achieved by simply removing the extra CPU(s), or may be best if you to create a new guest OS.

You need to analyse what's going on with the box as a whole, looking at CPU contention and %Ready, memory usage and IO latency and bandwidth. Although the basics of this can be quite easy to begin with I would suggest that you either get yourself trained in this matter (even a basic VCP4 course would help a lot) or get someone in to look at the problem who has done this before.

As for whether or not shutting down VMs will help, it could but it really does depend on what the issue is, you need to find that first.

Feel free however to add some actual details to your post showing any of this data and we'll take a look.