Dear Sajjan,
Usually, low cost Shared hosting servers have this problem of Low First byte time. It means the initial connection to the server is slow.
The usual solution is to shift to better hosting services. Like cloud hosting or managed hosting.
I see that you have Wordpress CMS. You could be better of transferring your site to a MANAGED WORDPRESS hosting platform. Namecheap.com & GoDaddy have some affordable plans for this. Also, check other providers.
All the best
Ajinkya Naik (India)