How To Synchronize Core Data with a Web Service – Part 1

This is a post by iOS Tutorial Team Member Chris Wagner, an enthusiast in software engineering always trying to stay ahead of the curve. You can also find him on Google+. A lot of apps that store their data in a remote database only work when an Internet connection is available. Think about Twitter or […]

Chris Wagner
Aug 22 2012 · Article · 50 mins

Version

Other, Other, Other

Learn how to synchronize Core Data with a web service!

This is a post by iOS Tutorial Team Member Chris Wagner, an enthusiast in software engineering always trying to stay ahead of the curve. You can also find him on Google+.

A lot of apps that store their data in a remote database only work when an Internet connection is available. Think about Twitter or Facebook – without an Internet connection, they don’t do much!

But it’s a much nicer (and faster) user experience if your app can work even without an Internet connection. And there’s good news – you can do this with caching!

The idea is you create a local cache of your data so you can access it whether the user is online or offline. And then when the user is online, you synchronize the cache and the remote database.

In this tutorial, you’ll learn how to do exactly that. You will create an app that stores its information locally with Core Data, and you will synchronize the records to a remote database when online. You can use the techniques you’ll learn in this tutorial with any web service, but to keep things simple you will use a popular web service platform called Parse to create and host your web service.

This tutorial assumes you have basic familiarity with Core Data and web services. If you are new to these topics, check out some of the other tutorials on this site.

So let’s get synchronized… 3, 2, 1, let’s go!

Getting Started

In this tutorial, you are going to create an app to help users solve a very common problem – remembering important dates! If you have ever forgotten a good friend’s birthday or your anniversary, raise your hand. Yep, thought so!

To keep the focus on the synchronization aspect, I’ve created a starter project for you. This starter project will get you to the point of running the application at a local level, where users can add and remove birthdays and holidays which will be stored locally in Core Data.

Your first order of business is to download run the starter project and get familiar with how it works.

Take a look through the app and make sure you understand how it’s currently working, how the Core Data database is set up, and how the view controllers flow together. Again, if you’re confused by anything here you might want to check out some of the other tutorials on this site first.

Once you feel comfortable with how the app is set up, keep reading to learn how to synchronize this local app to a remote web service!

Setup a free Parse account

Parse is a business that exists to provide the back end for your apps – i.e. the data tier of the servers and storage. Before such services existed, in order to get started with a connected application you would need to configure your own web server, database and probably write a whole lot of code. Not any more!

Parse is free for a rather substantial amount of usage. The free tier provides 1,000,000 API requests and 1,000,000 pushes per month, and provides you with 1GB of file storage. All that and other included services are more than enough to get your app established to the point where it will likely generate more than enough income to pay for the service.

This tutorial uses Parse so you will need to sign up. Parse is actively expanding their service and consistently improving their site. The registration process should be fairly straightforward for new users. Go do that step now before you continue with the rest of the tutorial. Already an existing Parse user? Then go ahead and log in!

Once you’ve created your account and logged in, create a new App for this tutorial from your Dashboard. Note that your App name in Parse can be whatever you want, but it will be referenced as CoreDataSyncTutorial, so you may prefer to use that name.

Once you are logged in and have your Parse App created, you will probably find that the browser is sitting at the Parse dashboard screen with the Overview tab selected. Select the Data Browser tab. The Data Browser will show all of your classes (analogous to the objects or entities that you see when editing your Core Data schema) – but so far there is nothing to see! Time to get busy and get your money’s worth out of this free service!

Press the “+” button to add a new class.

Enter “Holiday” as the name. Parse supports some predefined class types such as “User”, and offers support for common functions like logging in with name and password for those classes, but for now Custom is the default and is the right choice for your app.

Now add six new columns with the “+ Col” button; name, observedBy, image, wikipediaLink, details and date.

Press the +Col button to add a new column

Enter “name” as the name and set the type as String.

Continue adding the other columns using the +Col button. Use the list below to assist in creation of the columns.

name: String

observedBy: Array

image: File

wikipediaLink: String

details: String

date: Date

Just to save some heartbreak later, go back and verify that all of the columns (and their corresponding types) have been added later.

Repeat the above process for another class named “Birthday”. Again, use + to create the new class as “Custom” type and name it “Birthday”. Add the following columns to the new Birthday class:

name: String

date: Date

facebook: String

giftIdeas: String

image: File

Add Some Data

Time to get those all important holidays entered! Select the “Holiday” class, and click the +Row button to add a new record.

Let’s see; there’s Pi Day, Programmer’s Day, Towel Day, Talk Like A Pirate Day, Star Wars Day, and countless other incredibly important holidays — but to keep things simple for the sake of this tutorial, let’s stick to the examples below ;]

Fill out the fields with the following values. If the data browser is not letting you hit Return to confirm data entry in a field, it’s probably because you have some invalid data there. Make sure you enter the data exactly as shown, especially the more tricky formats like arrays.

It’s highly encouraged to add more records, but what you’ve entered already is enough for a sample set of data for the tutorial.

Okay! You’ve now created some content for your app – but now you need to get that data INTO your app! The next section shows you just how to do that!

Write an AFNetworking client to talk to the Parse REST API

AFNetworking is a class developed by Matt Thompson and Scott Raymond and is described as “a delightful networking library for iOS and Mac OS X.” It makes common tasks like asynchronous http requests a lot easier. As of this writing, AFNetworking does not use ARC, so if you manually add it to a project that is using ARC, the -fno-objc-arc compiler flag is necessary on all of its files. This has already been done for you in the tutorial project.

Note: This tutorial assumes some AFNetworking experience. If you have not used this library before, you should definintely have a read over Getting Started with AFNetworking before you go any further in this tutorial.

The first step to using AFNetworking in your app is to create a client that uses the Singleton pattern. Go to File\New\File…, choose iOS\Cocoa Touch\Objective-C class, and click Next. Enter AFHTTPClient for Subclass of, name the new class SDAFParseAPIClient, click Next and Create.

Open your interface file, SFAFParseAPIClient.h and add a new class method:

Note that in the code above, you need to insert the personal information generated by Parse: YOUR_APPLICATION_ID and YOUR_REST_API_KEY are unique to your app. Replace YOUR_APPLICATION_ID and YOUR_REST_API_KEY with the values from the Overview tab of the Parse project window.

It also creates three static NSString variables for the Parse API URL, your Parse API Application Id, and your Parse API Key, and implements the +sharedClient method which uses GCD to create a new instance of the class and store its reference in a static variable, thus becoming a Singleton.

Next import AFJSONRequestOperation.h in SDAFParseAPIClient.m:

#import "AFJSONRequestOperation.h"

Then override -initWithBaseURL: in SDAFParseAPIClient.m to set the parameter encoding to JSON and initialize the default headers to include your Parse Application ID and Parse API Key:

-GETRequestForClass:parameters: will return an NSMutableURLRequest used to GET records from the Parse API for a Parse Object with the class name ‘className’ and submit an NSDictionary of parameters. Acceptable parameters can be seen in the Parse REST API Documentation.

-GETRequestForAllRecordsOfClass:updatedAfterDate: will return an NSMutableURLRequest used to GET records from the Parse API that were updated after a specified NSDate. Notice that this method creates the parameters dictionary and calls your other method. This is merely a convenience method so that the parameters dictionary does not have to be generated each time a request is made using a date.

Okay! So you now have an AFNetworking client ready to go. But it’s not much good until you get the data synchronized! The section below will get you there.

Create a “Sync Engine” Singleton class to handle synchronization

Add another new Singleton class to manage all of the synchronization routines between Core Data and your remote service (Parse). Go to File\New\File…, choose iOS\Cocoa Touch\Objective-C class, and click Next. Enter NSObject for Subclass of, name the new class SDSyncEngine, click Next and Create.

In order to synchronize data between Core Data (your local records) and Parse (the server-side records), you will use a strategy where NSManagedObject sub-classes are registered with SDSyncEngine. The sync engine will then handle the necessary process to take data from Parse, and…uh…parse it (for lack of a better term!), and save it to Core Data.

Declare a new method in SDSyncEngine.h to register classes with the sync engine:

- (void)registerNSManagedObjectClassToSync:(Class)aClass;

Add a “private” category in SDSyncEngine.m with a property to store all of the registered classes and synthesize:

This method takes in a Class, initializes the registeredClassesToSync property (if it is not already), verifies that the object is a subclass of NSManagedObject, and, if so, adds it to the registeredClassesToSync array.

Note: It’s always preferable to write efficient code, but when it comes to synchronizing data with an online service, you want to get the most “bang for the buck” with each synchronization call. The Parse service may have a free tier, but that doesn’t mean it’s unlimited – and you want to make use of every call in the most efficient way possible! Plus keep in mind that every piece of data pulled over the mobile network counts against the user’s data plan. No one will want to use your app if it’s going to rack up excess data charges! :]

One main concern with keeping the data synchronized is doing it efficiently, so it doesn’t make sense to download and process every record each time the sync process is executed. One solution is to use a process generally known as performing a “delta sync”, meaning “only give me the new stuff, I don’t care about what I already know”.

Your delta sync process will be accomplished by looking at the “updatedAt” attribute on your Entities and determining which one is the most recent. This date will then be used to ask the remote service to only return records who were modified after this date.

This method iterates over every registered class, creates NSMutableURLRequests for each, uses those requests to create AFHTTPRequestOperations, and finally at long last passes those operations off to the -enqueueBatchOfHTTPRequestOperations:progressBlock:completionBlock method of SDAFParseAPIClient.

Notice that this method is not complete! Take a look at Comment 1 inside the success block for the AFHTTPRequestOperation. You will later add a method in this block that takes the response received from the remote service and saves it to disk. Now check out Comment 2; this block will be called when all operations have completed. You’ll later add a method here that takes the responses saved to disk and processes them into Core Data.

Awesome! You’ve written a ton of code already! You’re almost to the point where you’ll start seeing some progress in the application. However, at this point you’re lacking a way to start the whole sync process — which is the whole point of this app! :] But tread carefully – the manner in which the sync process is started is crucial, and it’s important to keep track of the status, as you do not want to start the sync process more than once.

You’ve implemented the -startSync method which first checks if the sync is already in progress, and if not, sets the syncInProgress property. It then uses GCD to kick off an asynchronous block that calls your downloadDataForRegisteredObjects: method.

Moving right along, you need to register your NSManagedObject classes and start the sync!Import the appropriate classes in SDAppDelegate.m:

This registers the Holiday and Birthday classes with the sync engine in -application:didFinishLaunchingWithOptions:; this pattern allows you to easily add other objects to the sync engine in the future, should you wish to extend the application! Scalability is always good! :]

Exciting stuff! Now it’s time to do something with this data; it’s just floating around, and you need to persist this data to local storage. The key concept in data transactions is to perform as many network operations as possible in a single batch, in order to reduce network traffic. A really easy way to accomplish this is to queue all requests and save the responses off to disk before processing them.

Add these three three methods to SDSyncEngine.m (beneath downloadDataForRegisteredObjects:) to handle file management:

The first two methods return an NSURL to a location on disk where the files will reside. The third is more specific to the application and the remote service; each response is saved to disk as its respective class name.

One interesting situation with the Parse API is that it will return values, which translate to NSNull objects when converted to an NSDictionary. Unfortunately, it is not possible to serialize an NSNull object to disk — you can’t save what is, essentially, nothing! Therefore, you must remove all of the NSNull objects before persisting your data.

You’ll take an optimistic approach here when persisting your data. You’ll attempt to same the response first, without scanning for NSNull objects. If that attempt fails, then scan for NSNull objects, remove any that are found, and try again. If THAT attempt fails, then all you can do is to fall back to standard error handling techniques, where you alert the user or report the issue. this tutorial won’t cover those error handling operations, but you can easily add your own if you so desire.

Next it’s time to modify downloadDataForRegisteredObjects – replace the placeholder comment 1 as follows:

You’ve replaced the comment with a much more useful action — writing the data to file! The NSLog goes away, and the -writeJSONResponse:toDiskForClassWithName: method is called.

Build and run your App! You’ll see that the files are saved to disk.

If you want to see the results of all your hard work, you can view the actual files by opening Finder, open the Go menu, choose “Go to Folder…” and enter ‘~/Library/Application Support/iPhone Simulator/’. (If you prefer to click through all the folders, hold down “option” when selecting Go in Finder and Library will reappear in the list of possible destinations.) From here, you will need to do some detective work!

First, determine which version of the simulator you are running. Open the appropriate folder for that simulator, then open Applications. Next you will likely see a number of folders with random names. Eek! Stay cool. An easy way to find the correct folder is to sort the folders by Date Modified and open the most recently modified folder. You will know you found the correct folder once you see “SignificantDates.app” in the folder.

Once you are in the correct App folder, open Library > Caches > JSONRecords. Here you will see the Holiday and Birthday files. You can open the files with Xcode to view them as they are simple Property List files. Hooray! If you see your data in the files, then you know your app is working! :]

At this point the sync process is finished! Even though it doesn’t do a whole lot right now, you still need to be aware when syncing is in progress, and when it is not. You already have a BOOL to track this, but you need to set it to NO at this point in order to stop sync. You’ll also want to know when the App is syncing for the first time, otherwise known as the initial sync. To track this information add the following @interface SDSyncEngine() in SDSyncEngine.m:

After -startSync add the method above which will be called when the sync process finishes.

Process remote service data into Core Data

Well, you have our data persisting to disk in a Property List format. But what you really want to do is to process it into Core Data. This is where you will be doing some heavy lifting and getting into the nitty gritty details. This is the part where the real magic happens!

To start off you will first need a way to retrieve the files from disk. Add -JSONDictionaryForClassWithName: in SDSyncEngine.m:

One caveat to the NSDictionary that -JSONDictionaryForClassWithName: returns is that the information you are interested in is will be in an NSArray with the key “results”. So to make things easier for processing purposes, add another method to access the data in the NSArray and spice it up a little to allow for sorting of the records by a specified key.

This method calls the previous method you implemented, and returns an NSArray of all the records in the response, sorted by the specified key.

You won’t really need the JSON responses that were saved to disk much past this point, so add another method to delete them when you’re finished with them. Add the following method above -JSONDictionaryForClassWithName:

In order to translate records from JSON to NSManagedObjects, you will need a few methods. First, you will need to translate the JSON values to Objective-C properties; the method you use will vary based on the remote service you are working with. In this case, you are using Parse which has a few “special” data types. The data you’ll be concerned with here are Files and Dates. Files are returned as URLs to the file’s location, and Dates are returned in the following format:

{
"__type": "Date",
"iso": "2011-08-21T18:02:52.249Z"
}

Since the date is in the format of a string, you will want some methods to convert from a Parse formatted date string to an NSDate and back to an NSString. NSDateFormatter can help with this, but they are very expensive to allocate — so first add a new NSDateFormatter property that you can re-use.

The first method -initializeDateFormatter will initialize your dateFormatter property. The second method -dateUsingStringFromAPI: receives an NSString and returns an NSDate object. The third method -dateStringForAPIUsingDate: receives an NSDate and returns an NSString.

Take a little closer look, there, detective — the second and third methods do something a little strange. Parse uses timestamps in the ISO 8601 format which do not translate to NSDate objects very well, so you need to do some stripping and appending of the milliseconds and Z flag (used to denote the timezone). (Oh standards…there are so many wonderful ones to choose from!) :]

This method accepts a value, key, and managedObject. If the key is equal to createdDate or updatedAt, you will be converting them to NSDates. If the key is an NSDictionary you will check the __type key to determine the data type Parse returned. If it is a Date, you will convert the value from an NSString to an NSDate. If it is a File, you will do a little more work since you are interested in getting the image itself!

To get the image, send off a request to download the image file. It is important to note that downloading the image data can take a considerable amount of time, so this may only work efficiently with smaller data sets. Another solution would be to fetch the image data when the record is accessed (lazy loading), but it would only be available if the user has an Internet connection at the time of lazy loading.

If the data type is anything other than a File or Date there is no way to know what to do with it so set the value to nil. In any other case you will simply pass the value and key through untouched and set them on the managedObject.

Next, add methods that create an NSManagedObject or update an NSManagedObject based on a record from the JSON response to SDSyncEngine.h:

newManagedObjectWithClassName:forRecord: accepts a className and a record, using this information it will create a new NSManagedObject in the backgroundManagedObjectContext

-updateManagedObject:withRecord: accepts an NSManagedObject and a record, using this information it will update the passed NSManagedObject with the record information in the backgroundManagedObjectContext

You’re heading into the home stretch! Just two more methods before you tie it all together in a method that actually processes the JSON data into Core Data. Add these methods right after -setValue:forKey:forManagedObject:

-managedObjectsForClass:withSyncStatus: returns an NSArray of NSManagedObjects for the specified className where their syncStatus is set to the specified status,

-managedObjectsForClass:sortedByKey:usingArrayOfIds:inArrayOfIds: returns an NSArray of NSManagedObjects for the specified className, sorted by key, using an array of objectIds, and you can tell the method to return NSManagedObjects whose objectIds match those in the passed array or those who do not match those in the array. You’ll read more about this method later on.

Now to put all of these methods to use!

- (void)processJSONDataRecordsIntoCoreData {
NSManagedObjectContext *managedObjectContext = [[SDCoreDataController sharedInstance] backgroundManagedObjectContext];
//
// Iterate over all registered classes to sync
//
for (NSString *className in self.registeredClassesToSync) {
if (![self initialSyncComplete]) { // import all downloaded data to Core Data for initial sync
//
// If this is the initial sync then the logic is pretty simple, you will fetch the JSON data from disk
// for the class of the current iteration and create new NSManagedObjects for each record
//
NSDictionary *JSONDictionary = [self JSONDictionaryForClassWithName:className];
NSArray *records = [JSONDictionary objectForKey:@"results"];
for (NSDictionary *record in records) {
[self newManagedObjectWithClassName:className forRecord:record];
}
} else {
//
// Otherwise you need to do some more logic to determine if the record is new or has been updated.
// First get the downloaded records from the JSON response, verify there is at least one object in
// the data, and then fetch all records stored in Core Data whose objectId matches those from the JSON response.
//
NSArray *downloadedRecords = [self JSONDataRecordsForClass:className sortedByKey:@"objectId"];
if ([downloadedRecords lastObject]) {
//
// Now you have a set of objects from the remote service and all of the matching objects
// (based on objectId) from your Core Data store. Iterate over all of the downloaded records
// from the remote service.
//
NSArray *storedRecords = [self managedObjectsForClass:className sortedByKey:@"objectId" usingArrayOfIds:[downloadedRecords valueForKey:@"objectId"] inArrayOfIds:YES];
int currentIndex = 0;
//
// If the number of records in your Core Data store is less than the currentIndex, you know that
// you have a potential match between the downloaded records and stored records because you sorted
// both lists by objectId, this means that an update has come in from the remote service
//
for (NSDictionary *record in downloadedRecords) {
NSManagedObject *storedManagedObject = nil;
// Make sure we don't access an index that is out of bounds as we are iterating over both collections together
if ([storedRecords count] > currentIndex) {
storedManagedObject = [storedRecords objectAtIndex:currentIndex];
}
if ([[storedManagedObject valueForKey:@"objectId"] isEqualToString:[record valueForKey:@"objectId"]]) {
//
// Do a quick spot check to validate the objectIds in fact do match, if they do update the stored
// object with the values received from the remote service
//
[self updateManagedObject:[storedRecords objectAtIndex:currentIndex] withRecord:record];
} else {
//
// Otherwise you have a new object coming in from your remote service so create a new
// NSManagedObject to represent this remote object locally
//
[self newManagedObjectWithClassName:className forRecord:record];
}
currentIndex++;
}
}
}
//
// Once all NSManagedObjects are created in your context you can save the context to persist the objects
// to your persistent store. In this case though you used an NSManagedObjectContext who has a parent context
// so all changes will be pushed to the parent context
//
[managedObjectContext performBlockAndWait:^{
NSError *error = nil;
if (![managedObjectContext save:&error]) {
NSLog(@"Unable to save context for class %@", className);
}
}];
//
// You are now done with the downloaded JSON responses so you can delete them to clean up after yourself,
// then call your -executeSyncCompletedOperations to save off your master context and set the
// syncInProgress flag to NO
//
[self deleteJSONDataRecordsForClassWithName:className];
[self executeSyncCompletedOperations];
}
}

This method must be called where Comment 2 sits as a placeholder in the downloadDataForRegisteredObjects method. When the HTTP request operations are completed, returned data is written to Core Data, and your app can then access them as required.

This is it, kids! This is the final downloadDataForRegisteredObjects method!

Run the project and see your records created in Parse come in to your App!

Manually trigger sync with remote service

Right now, your App will automatically sync when the user opens the App. This is generally an acceptable practice; however some users may want to refresh the dataset without leaving the App. You’ll next implement a refresh button on both the Holiday table view and Birthday table view in order to accomplish this.

Start by opening Storyboard.storyboard, locating the Holidays Date Table View Controller and dragging a Bar Button Item onto the top left side of the navigation bar. Once you have dragged the bar button item onto the nav bar, change its Identifier attribute to “Refresh” as seen below:

Now open the Assistant editor (the bow tie button in Xcode) and make sure SDDateTableViewController.h opens in the editor. Holding down the CTRL key, click and drag from the Refresh button to your interface to add a new IBAction named “refreshButtonTouched”:

Do this same process again, but this time instead of creating an Action, create an Outlet for the button and name it ‘refreshButton’. The outlet should be a strong reference, not a weak one — otherwise when the button is replaced by the activity indicator, it will be nil when the time comes to replace the button on the navigation bar.

Now select the Refresh button in your Storyboard, and while holding the Option/Alt key, click and drag it to your Birthdays Date Table View Controller to copy it, by copying you will also copy the outlets you already set up. Once you copy the button, with the Assistant editor still open you can hover the circles in the gutter next to your IBAction and IBOutlet to visually see the referenced outlets in the Storyboard, both buttons should highlight as shown below.

Open SDDateTableViewController.m and add an import for SDSyncEngine.h:

#import "SDSyncEngine.h"

Next complete the -refreshButtonTouched: method that was created for you during the Storyboard editing you just did, and add the following methods at the bottom of your class above @end:

-checkSyncStatus will ask the SDSyncEngine singleton if the sync is in progress. If so, it will call -replaceRefreshButtonWithActivityIndicator which does what it says,which is replace the refresh button with a UIActivityIndicatorView. Otherwise, the method will remove the UIActivityIndicatorView by replacing it with the refreshButton.

You will also implement -observeValueForKeyPath:ofObject:change:context: in order to observe changes in SDSyncEngine. To do so you will need to register for those notifications:

Update -viewDidAppear: and -viewDidDisappear: to look like the above, also notice when the view appears we check the sync status immediately.

Now when you build and run the App, you will see an activity indicator while the sync is in progress and a refresh button when the sync is inactive. Touching the refresh button fires off the sync engine.

Contributors

Chris Wagner

Chris Wagner is currently working as an Independent Contractor primarily with MartianCraft focused on iOS development. Prior to...

Author

This is a post by iOS Tutorial Team Member Chris Wagner, an enthusiast in software engineering always trying to stay ahead of the curve. You can also find him on Google+. A lot of apps that store their data in a remote database only work when an Internet connection is available. Think about Twitter or […]