1. Expensive. You know you have to provide for all those womens, yes?
2. Diminishing returns. After you reach a certain number of people, there's not a whole lot that an extra body can really do for you. A massive harem is nice to look at, but the novelty wears off quickly once you realize you can do more than look. Combine this with #1, and you're looking at a very bad investment.
3. Too much variety leads to a lack of mastery. You know how people tell you to pick a couple champions and learn to play them really well? Same idea here. If wife #47 is only really with you once a month or so, she's not
4. Security. Let's be honest... Your harem probably isn't going to be thrilled with the arrangement, especially as the size of your harem increases. Last thing you want is a jealous harem girl ganking another one of your harem girls. Or worse, deciding to try to take you out. And your harem is something that other people will covet, so you're always going to have to worry about people trying to steal your harem girls away. Trust me, you do NOT want to have to face the rest of the guys on the golf course after they find out that one of your palace guards ran off with your favorite harem girl.

So yeah... Harem is a bad idea. You should really reconsider.

What you really want is to live the Charlie Sheen lifestyle, or the Hugh Hefner lifestyle. Max yourself out at 3-4 (Hef went a little crazy there for a while with 7 girlfriends, but quickly realized he needed to scale that back to the 3 he really cared about), make it clear that you love all women in general, and that the women you're with are there by choice. Sheen and Hef are all about female empowerment. Get a reputation for that sort of thing and you'll wind up with a nice healthy rotation. It will feel just like a harem, but you keep your expenses in check (more money to spend on your now legal weed), the women get enough attention to "master their champion", and you don't have to suspiciously guard them all the time.