We stop in Victoria two years ago and everyone had to pass thru customs who ever left the ship. We had to show our passports to the Canadian customs. Our friend who has the same problem as you goes to the US almost 5 times a year and never was stopped till the last time about his record. The US custom agent question him 2 hours before letting him proceed. It totally depend on the person at custom. There is no right answer. We drive over to the US 4 or 5 times ourself and twice they have given us a hard time for no reason. My wife and I have no record in our 60 years. We never have any issues coming back with the Canadian customs.