This use of temple is a separate etymology than the word "temple" for "place of worship". Both come from Latin, but the word for the place of worship comes from templum, whereas the word for the part of the head comes from Vulgar Latin *tempula, modified from tempora, plural form ("both temples") of tempus, a word that meant both "time" and the part of the head. Due to the common source with the word for time, the adjective for both is "temporal" (both "pertaining to time" and "pertaining to the temple").