Achievement-wise, only the first fight between them really mattered. The 2nd and the 3rd was between two "has-beens", of a lot less significance.

Norton was still seen as a poor boxer in 1981, so even if you transfered those voters into the 1980's, they'd still be forced to go see a doctor. Glen Johnson was a journeyman, good fighter, but not great. His KO of Jones is like Leon Spinks' decision over Ali, on that level.
Let's pick another one, Montell Griffin, if he were a heavyweight, would have a lot more reason to be considered a great, much better boxer than Norton, and with better achievements. He's not getting to IBHOF any time soon, possibly never.
Norton lost to Holmes. If you want to praise fighters based on Holmes winning very closely or even controversially over them, that'd make a lot of 1980s heavyweight tomato cans ATGs. Norton's win over Young was a robbery.
Whenever Ali chose to dance around on his toes, Norton was clueless at what to do. It's only that Ali could not keep moving the whole fight, only had done these during several rounds, that made the fights close. This lack of ring generalship on the part of Norton makes me confident that younger Ali would have gained an easy victory over Norton.

Yes, for P4P (ie ATG, regardless of weight) ratings I compare fighters not only for how great they were at some particular division, but who has more depth overall, who beat better opponents and in much greater quantities. Limiting the rating to heavyweight division, Ali is a great, same as the other names you mentioned. But compared to a lot of ATG smaller fighters Louis and Ali are light years behind, like I said. Had Louis beaten Jeffries, Ali and Lewis, he would have deserved to join the company of Gans and Leonard. But the guys he had beaten are laughable compared to the guys both these lightweights had beaten. To put Ali above them is a horrible disgrace. Head to head both were also much better than Ali.

You saw the words "head to head" right? You quoted them. What are you bringing Duran's and Ali's resumes for? Head to head, make Ali a lightweight, or Duran a heavyweight, while keeping their styles and fighting qualities, I'd bet my house on Duran.