long term prometheus metrics

Description

data retention on the primary prometheus has been expanded to 30 days, which is nice, but that's not enough. create another (a third, technically, but a second in this cluster) prometheus server that would scrape *all* metrics off the *first* server, but at a different sampling rate so we can keep metrics for a longer, possibly multi-year timeline.

Review the storage requirements math in #29388 and compare with reality.

This, obviously, is a followup to the general prometheus setup ticket in #29389.

So how long do we want to keep that stuff anyways? I like the 15 minutes 5 year plan, personnally (20GB) although I *also* like the idea of just shoving samples every 5 minutes like we were doing with Munin, which gives us 12GiB, or 60 GiB over five years...