We study the empirical properties of realized volatility of the E-mini S&P 500 futures contract at various time scales, ranging from a few minutes to one day. Our main finding is that intraday volatility is remarkably rough and persistent. What is more, by further studying daily realized volatility measures of close to two thousand individual US equities, we find that both roughness and persistence appear to be universal properties of volatility. Inspired by the empirical findings, we introduce a new class of continuous-time stochastic volatility models, capable of decoupling roughness (short-term behavior) from long memory and persistence (long-term behavior) in a simple and parsimonious way, which allows us to successfully model volatility at all intraday time scales. Our prime model is based on the so-called Brownian semistationary process and we derive a number of theoretical properties of this process, relevant to volatility modeling. As an illustration of the usefulness our new models, we conduct an extensive forecasting study; we find that the models proposed in this paper outperform a wide array of benchmarks considerably, indicating that it pays off to exploit both roughness and persistence in volatility forecasting.