Swift data is persistent in one table view, but not when switching between other view controllers

I'm looking to implement a contacts list into my app. I have the add contact working and it actively uses NSCoding for writing to the disk for the contacts. It will update properly between the list of contacts and the new contact view, but doesn't when I switch from another view.

Do I need to prepareforsegue when coming from the non-contact views in a way that prepares the data? Or do I need to create my Set in the first view and pass references through that when going between views. As of now it only updates data when I do my unwindsegue from the add contact view controller.