Generally guys don't show it...but when they do show how scared they are of losing you then it is practically required to spend more time. He's probably just feeling a little uncomftorable when he's not with you because he's afraid of losing you to another guy.