So regardless of your assigned number, you can hang on the ship until the end?

Yup The assigned number is to let you know you may get off now if you want to. We went on a cruise with my MIL, she had the last color called and we had one of the earlier colors called. We just hung out and played cards until like 10 minutes after her color was called the line was so much shorter.