SVG files are vector-based, and Photoshop is primarily pixel-based. Photoshop isn't as good with handling SVG files, so I would recommend exporting PNG files from XD if you want to bring them into Photoshop.