We present a simple method for the co-encapsulation of gold nanostars and iron-oxide nanoparticles into hybrid colloidal composites that are highly responsive to both light and external magnetic fields. Self-assembly was driven by hydrophobic interactions between polystyrene capped gold nanostars and iron oxide nanocrystals stabilized with oleic acid, upon addition of water. A block copolymer was then used to encapsulate the resulting spherical colloidal particle clusters, which thereby became hydrophilic. Electron microscopy analysis unequivocally shows that each composite particle comprises a single Au nanostar surrounded by a few hundreds of iron oxide nanocrystals. We demonstrate that this hybrid colloidal system can be used as an efficient substrate for surface enhanced Raman scattering, using common dyes as model molecular probes. The co-encapsulation of iron oxide nanoparticles renders the system magnetically responsive, so that application of an external magnetic field leads to particle accumulation and limits of detection are in the nM range.