RX 5700 XT Fuzzy/Noisy Artifacts on White Backgrounds

So, I rolled back to 19.12.1 using DDU and the fuzziness on boot problem is gone. I believe 19.12.1 is the last version before they introduce Adrenalin 2020 (19.2.2). THE PROBLEM IS NOW FIXED!

I have a SAPPHIRE NITRO+ RX 5700 XT (not OC'ed in any way with afterburner)

Driver version: 20.3.1

CPU: Ryzen 5 3600 (no OC, default bios settings)

Chipset: X370 (ASUS PRIME X370-PRO Ver.5220)

Windows 10 1909 (most up to date)

Every time when I boot up Windows 10, there's this layer of static or fuzziness on my screen when I am looking at a white background. The easiest way to tell is to use win+e to get a window open with white background, or go to Google's homepage, but it's basically apparent on all white colors.

The only way to fix this is to unplug my DP cable and plug it into the adjacent one. This works 100% of the time and only when I switch output port, if I unplug and plug it back into the same port, the problem persist.

Another way of "fixing" it is to disable and re-enable my 5700 XT under Device Manager. But I tend not to use it b/c it bricks my start menu and it won't open.

I have also tried to use HDMI, but the same problem shows up after reboot. Can be "fixed" by unplugging and plugging back into an adjacent HDMI output.

Things I've tried:

- Clean install of Windows 10 1909 (didn't fix)

- Install the latest chipset driver (did fix for a while, but after couple reboot the fuzziness came back)

- Rollback to previous drivers (only 19.12.2 for now, didn't work, but also didn't use DDU, might try out other versions)

- Unplug all PCIe cards only 5700 XT in place (didn't fix)

- Different PCIe slot (didn't fix)

My assumptions:

- Doesn't seems to be a hardware problem given that:

- chipset driver update did fix it for a while

- when I got the card back in Jan 2020, I didn't notice this consistently, and now I know for every reboot it comes back, it makes me believe it has something to do with the GPU driver?

What bothers me is how easily this is fixed, just by switching output, also when I was in the windows 10 setup/installation the problem wasn't there.