As you can see, when loaded for the first time, it is taking a long time to wait. After that, when I reload the website, it loads much faster. After that I do not open the site for an hour approx. Again, after checking after an hour or so, it is loading slow again in the first run.

That is usually a cache somewhere on the server (or CDN). The actual site itself is slow to build the page (more often than not a slow database) but the generated page is cached so it can be served quickly after that.

It could also be other caches in the application (database query cache, filesystem caches, etc) but more often than not it is a page cache.