We just take our driver's license and our ship card. The only time they told us to take our passports was in Trinidad. We were met with their local security upon reboarding and they actually did take our passports (not to keep, they just wanted to verify). Every other port, our license and ship card were fine.