How are you switching it off? A full boot up from a no/dead battery is quite long, but the device is not designed to start from this state frequently.
The soft power off (holding down the power button on the top) puts it into more of a standby and shouldnt take more than a couple seconds.