Does changing the stock CPU cooler for an after-market one make a difference?

Whether buying a pre-built system or constructing your own, the stock CPU cooler isn't usually much to shout about.
The cooler that came with your PC will does the job. It has to, otherwise, you're buying a faulty product. But doing the job and doing a good job are different things entirely. The Alienware Aurora R5 I picked up and reviewed recently came with a pretty puny air cooler over its Core i7 6700 CPU. Not particularly impressive for a high-spec pre-built desktop PC.
Nevertheless, it did the job fairly well. It wasn't particularly loud and overly high temperatures didn't seem to be a thing. But I still decided to take a punt and swap it to see if there was any real improvement.
For science.
And there's no liquid involved here, we're talking air cooler to air cooler.