Meaning "remoteness, space between things or places" is late 14c. The figurative sense of "aloofness" is the same as in stand-offish. Phrase go the distance (1930s) seems to be originally from the prize ring, where the word meant "scheduled length of a bout."