What are the drawbacks if I don't share these files? None, from my experience. When doing a fresh checkout of a project, IDEA asks whether I'd like to create a new project. Yes please, and all files are generated correctly.

The only annoyance we have had at the company from time to time is when someone accidentally adds an .iml file to the VCS. There's a feature in IDEA to stop it from asking:

Settings -> Version Control -> Ignored Files

This setting has to be applied in every project separately. And thus it's not very helpful in this case. Because every new project I check out, IDEA asks me whether I want to add the .iml file to version control (one by one), and I can just tell it to stop asking:

Conclusion

For the time being we keep IDEA files out of the VCS. Our projects are usually multi-module maven, with lots of iml files. Your mileage may vary.