Maybe it's not so much breaking the promise as it is keeping her close enough to just be friends, but distant enough that she won't be in danger if a villain found out his identity or anything like that.
I don't know, that's just my way of thinking about it.