In summary,
these results suggest that absolute pitch is a primary perceptual mode
that is heavily superseded by relative pitch (probably in the course
of language acquisition). Early musical training or learning a
tonal language like Thai or Japanese may help to prevent this edging
out-process, with the consequence that certain subjects retain the
ability to perceive absolute pitch throughout life. Verbal
categorizations of notes may be helpful in this respect, but it would
be misleading to take them for the main underlying
cause.

A paper by Sandra Trehub suggests that infants are also sensitive
to relative pitch:

In searching for this ref, I just found
another of Trehub's demonstrating the ability of non-musicians to
discriminate recordings of instrumental themes from popular TV shows
in their correct key, from those transposed up or down by only one or
two semi-tones. They conclude that "Adults' reportedly poor
memory for pitch is likely to be a by-product of their inability to
name isolated pitches."