Nginx (I assume) sometimes adds 5000ms to response time

There is a weird thing I noticed on one of the servers that act as proxy that's bugging me for hours and I can't figure it out. Some requests get served to client exactly 5000ms after the content is fetched from upstream

It happens for a small percentage of requests and it usually comes in tiny "bursts" - it means I can see 5 such requests being served in a few seconds (its also higher possibility of higher delay but not exactly 5s in that timeframe) then everything is OK for a few minutes. I searched everywhere if there is some timeout configured in configs to 5 seconds or if there is some default nginx value set to 5 seconds but can't find anything. It happens on different cleints/networks.