In this context, there isn’t much difference between the two. “Trójkę” sounds a tiny wee bit more colloquial, but it’s nothing to worry about in actual usage.

“Trójka” might be safer to use, as it doesn’t seem to differentiate between genders. So you can say “trójka dzieci” even if you know that they’re all male, while “troje dzieci” might sound a bit strange in this case (as it evokes a mixed-gender group).

Take the above paragraph with a grain of salt though—Polish collective numerals can be problematic even for native speakers…