Yeah, Alfred didn't come from Gotham so he doesn't have that attachment at all. The Wayne legacy keeps him there.

__________________"Lets make one thing very clear here - Nolan's films are as faithful an adaptation as there is. It pays homage to its source material, remains true to its characters and above all else places the story first and foremost." - jmc

You could argue that Bruce's attachment to Gotham was based on the Wayne family as well. His strive to fix the city as both Bruce Wayne and Batman was heavily influenced by Thomas Wayne's philanthropy. He was always attempting to emulate his father in one way or another, with the most transparent being Bruce's fusion reactor project directly echoing his father's monorail. So in the end when he escapes Gotham he "dies" as Bruce Wayne as well as Batman, letting go of his burden to his family's legacy and thus the city itself.