I have both (got both cameras at their respective introduction), and keep them for different purposes. If you're into war photography or sports, then yes I'd say it's worth the upgrade. If you shoot indoor stuff or performances, then hell no, it would be the wrong move. Without saying more about what you're doing, hard to tell

1Dx = better ergonomic, faster fps, dual CF slot. But I don't know if those features are enough for me to upgrade.

I was debating to pick up a 1D MKIV to go with my 5D3, but I've read that the image quality isn't up to par with the 5D3. That's why I'm thinking about the 1Dx and maybe pick up another a 5D2 later on.

It's from the Digital Picture com review.. And the things about the 1DX is like he wrote in his review. You can change focus screens. And it has Digic 5+ and so on. You can read about it on the Canon website also.
In the menu you can even change the settings depending on what focusing screen that you use. And here is from Canon CPN about the digic
The EOS-1D X deploys two DIGIC 5+ processors that offer a variety of benefits including fast image processing to deliver 18.1 Megapixel images at up to 12fps as standard, and up to 14fps with super High Speed continuous.Enlarge image

To me, it's one of those question. Would you rather have 2 cameras or just one. I would basically have to give up two cameras. As a hobbyist I know one camera is enough, but sometimes I get wedding gigs and I just need that second camera. Changing lens wouldn't be practical during a wedding. It's really a tough decision. One camera can do it all vs. two camera that can do almost the same.

I love my X even though much of my shooting would be well handled by the 5D3 I have earlier. I have a 6D on order as my second body so i will have a lighter body that has the WiFi and GPS when I want those features. I want to be able to set the camera up for wildlife shots with remote control. I could have bought the 1DX wifi but it seemed like having a second body with that built in made sense for me.