Big solution suggestions

We have a big .NET solution (a company library framework) with multiple library DLL's, some EXE tools, and static files like MSBuild targets and schema XSD's. The whole thing is version-controlled in its entirety. It has been xcopy-deployed so far, all
compiled files+static files in one directory, but now we want to pack this with NuGet.

The solution has external NuGet dependencies, for example, there's a library which depends on Entity Framework. Not all projects use all components of this framework. It is possible that a project doesn't use Entity Framework and doesn't use our EF library
either. So we don't want to pull in the Entity Framework dependency to all our projects. It is not a problem with xcopy deployment: even if our EF library is there, we just won't use it.

The idea so far is to package each libary DLL as a separate package, one additional package for all the targets files, and maybe a separate package for the XSD's or put them along with a DLL, not sure yet. I found two problems with this so far.

Whenever a component changes, we have to release a new version of whole solution. So it's possible that only one component changes, but new NuGet packages of everything will be created. It's not trivial to compare if there is actually any change between two
versions of a NuGet package. So it will cause a bit of a version pollution.

The other problem is that there are some EXE files in the solution. There would be a single package with a single EXE tool in it, and this will be treated as a solution-level package. This EXE depends on some library DLL's, which will be in separate packages.
I guess that the EXE will not find the DLL's from other packages, so we have to package the library DLL's inside this package. Which means that a single library DLL will be packaged inside its own library package plus several other tool packages. Another case
of wasted space.

If we wanted to version-control each package separately, that would cause a lot of rework, because there are many dependencies between everything. If we wanted to put everything into one package, because it would pull in too much dependencies. Do you have some
ideas about this?