They don't have to steal your phone, they just have to get your phone number. So many stupid apps and whatever store the number, then get hacked. Think about Pannera Bread, which had a breach a year ago. Most of the data was just names and phone numbers, which seems benign on the surface, but you can do a lot with that info.