I think the statement is a bit too strong. I would argue that a newborn human can not understand any metaphysical concept, since he or she lacks the ability to use language. Thus you are not born Atheist, you are born neither theist nor atheist.