I build systems like that all the time, starting with command line installation and adding only what I want. Works just fine. I've built several Cinnamon installations this way. Cinnamon is forked off GNOME 3 Shell, so there's still plenty of GNOME underneath, but that's by design and nothing to worry about.