I've only brewed a few beers and I've put my hops in a nylon bag during the boil to cut down on the trub. However, I just brewed a Pliny the Elder clone and there were a LOT of hops in that bag. Did I limit the effectiveness of the hops by not letting them have more "room" during the boil?

Maybe, that's why I built myself a hop spider. Google it. $10 build for peace of mind. You can use one or five gallon paint strainer bags from a hardware store. The 5 gallon will give the hops a ton of room to float around in the boil.

How much hops? What size bag? If its a buttering hops being boiled for 60 mins, probably won't matter. A late addition hops in a tight bag might be affected. In n end, your more likely to be affected by a high boil gravity than by hops in a bag.