Maybe he sees all the ways this could reduce your time and opportunities together if it were frequent, moreso than a local lover would, and the thought of missing out on time and experiences with you saddens him, and then it saddens him further that the same thought doesn't make you sad, because he thinks that means you don't care about him as much. Maybe he's expressing his sadness as anger.