And even then sometimes it won't change or it changes back. It's a PITA.

If it doesn't change, force quit the Music.app and restart it.

To avoid streaming music while I was not on wifi prior to switching to unlimited data with my carrier, I had streaming over data disabled in Settings > Music > Cellular Data. That way it simply skipped songs that were not locally stored on my iPhone.