In general it’s just fine to leave the SD card in over power offs. The system will recognize that the version is the same as what’s loaded and will skip the reload. (There are debug versions which load every time but they shouldn’t be in the wild.)