The Buffy actor talks about arriving back home on his brother’s birthday, 13 days after he was born, how he missed his family when he moved to LA and how his father tried to shield him from disappointment