Yes, he was. It is worth noting that if Federer hadn't fallen to #4 for 6 weeks in 2011, he probably would have passed Connors for consecuive weeks at #3. 432 + 6 weeks (he was #4) + last 6 weeks of 2011 (ranked #3) + 52 weeks (all of 2012) = 496.

He would have to be in the top 3 for another 12 weeks in 2013 to pass this record.