Shazron Abdullah
added a comment - 18/Mar/12 18:00 Looks like there is a NSUserDefaults key "WebKitLocalStorageDatabasePathPreferenceKey" that points to the old db: http://stackoverflow.com/questions/9679163/why-does-clearing-nsuserdefaults-cause-exc-crash-later-when-creating-a-uiwebview

Summary:

The error can be replicated only when on iOS 5.1, and upgrading an app to an updated version. On a new app install, you won't see this.

The bug is that the "WebDatabaseDirectory" and "WebKitLocalStorageDatabasePathPreferenceKey" values from NSUserDefaults (which are effectively read from Library/Preferences/[CFBundleIdentifier].plist) are incorrect, they refer to the previous version's app path.

What's probably happening is, during an app upgrade, the app was moved/renamed to a new GUID, but these two plist values were not updated accordingly. The renaming of the app's application bundle to a new GUID might be new behaviour with iOS 5.1 app upgrades.

Background:

The first clue we had was this error in the console log:
deny file-write-create /private/var/mobile/Applications/43A38BD0-439A-4998-A51D-110C6E05AF7C

The simplest explanation: permissions error, but what file? But all files in our app bundle we should have permissions to already.

On application launch, we printed out the value for "WebKitLocalStorageDatabasePathPreferenceKey" from NSUserDefaults, and compared it to the application's bundle path. The value for the key should reside entirely inside the application's bundle path, since this is where Webkit stores the localStorage database (similarly for WebSQL, the location for the database is through the "WebDatabaseDirectory" key).

Everytime we see the permissions error, we see that the value for "WebKitLocalStorageDatabasePathPreferenceKey" contains a path OUTSIDE of our application bundle. Bingo!

Proposed Fix:

On app launch on an iOS 5.1 system, we test the keys "WebDatabaseDirectory" and "WebKitLocalStorageDatabasePathPreferenceKey" from NSUserDefaults, and if the paths fall outside of the application bundle's path - we change the paths to reflect the correct paths within the application bundle.

We then commit the new path changes by reading then writing from the Library/Preferences/[CFBundleIdentifier].plist file, then synchronize NSUserDefaults to pick up the new changes.

Shazron Abdullah
added a comment - 20/Mar/12 06:45
With the help of Justin Harrison, the problem has been diagnosed.
Summary:
The error can be replicated only when on iOS 5.1, and upgrading an app to an updated version. On a new app install, you won't see this.
The bug is that the "WebDatabaseDirectory" and "WebKitLocalStorageDatabasePathPreferenceKey" values from NSUserDefaults (which are effectively read from Library/Preferences/ [CFBundleIdentifier] .plist) are incorrect, they refer to the previous version's app path.
What's probably happening is, during an app upgrade, the app was moved/renamed to a new GUID, but these two plist values were not updated accordingly. The renaming of the app's application bundle to a new GUID might be new behaviour with iOS 5.1 app upgrades.
Background:
The first clue we had was this error in the console log:
deny file-write-create /private/var/mobile/Applications/43A38BD0-439A-4998-A51D-110C6E05AF7C
The simplest explanation: permissions error, but what file? But all files in our app bundle we should have permissions to already.
On application launch, we printed out the value for "WebKitLocalStorageDatabasePathPreferenceKey" from NSUserDefaults, and compared it to the application's bundle path. The value for the key should reside entirely inside the application's bundle path, since this is where Webkit stores the localStorage database (similarly for WebSQL, the location for the database is through the "WebDatabaseDirectory" key).
Everytime we see the permissions error, we see that the value for "WebKitLocalStorageDatabasePathPreferenceKey" contains a path OUTSIDE of our application bundle. Bingo!
Proposed Fix:
On app launch on an iOS 5.1 system, we test the keys "WebDatabaseDirectory" and "WebKitLocalStorageDatabasePathPreferenceKey" from NSUserDefaults, and if the paths fall outside of the application bundle's path - we change the paths to reflect the correct paths within the application bundle.
We then commit the new path changes by reading then writing from the Library/Preferences/ [CFBundleIdentifier] .plist file, then synchronize NSUserDefaults to pick up the new changes.

Urs Zimmermann
added a comment - 20/Mar/12 07:58 Shazron you are the man! Thank you for your analysis.
Unfortunately I am a phonegap/cordova user and not an iOS developper.
Do I get this right: Your proposed fix would be a solution for cordova itself? When would that be ready? (My app update is due in the next week...)

If it works, this will be incorporated into Cordova 1.6.0, and also the plugin for older versions (see CB-330). We're targeting a RC release on Mar 26th.
The proposed fix hasn't been fully tested yet, hopefully after testing it can be accepted.

Shazron Abdullah
added a comment - 20/Mar/12 08:02 If it works, this will be incorporated into Cordova 1.6.0, and also the plugin for older versions (see CB-330 ). We're targeting a RC release on Mar 26th.
The proposed fix hasn't been fully tested yet, hopefully after testing it can be accepted.

Shazron Abdullah
added a comment - 20/Mar/12 15:20 Filed a bug with Apple, you should too, to raise the visibility of this issue.
Reported to Apple: rdar://11081309
Copied here: http://openradar.appspot.com/radar?id=1608403

Daniel
added a comment - 21/Mar/12 14:27 I tried installing an app using localstorage (let's call it v1) without the CB-330 fix, then upgraded to a new version of the app with the CB-330 fix (v2).
The data from v1 is still visible in v2. Now I add some additional data, I kill the app and restart it. The old data is still there, however the newly added data is gone.
I don't understand why this happens. Perhaps because the code is copying the backup db over the newly modified version?

It seems to allow my app access to the database after an update finally, but I still get the "SECURITY_ERR: DOM Exception 18" database access error when the plugin applies the "verifyAndFixDatabaseLocations" function on the first install of the app over my old app. Once the locations of the databases are fixed and the app has been restarted, it works perfectly, but it seems as though the locations aren't being applied until the app is restarted.

Any suggestions as to how this can be fixed? I'd really like to finish the fix to my app ASAP.

Mike Newman
added a comment - 21/Mar/12 15:53 Thanks alot for the fix!
It seems to allow my app access to the database after an update finally, but I still get the "SECURITY_ERR: DOM Exception 18" database access error when the plugin applies the "verifyAndFixDatabaseLocations" function on the first install of the app over my old app. Once the locations of the databases are fixed and the app has been restarted, it works perfectly, but it seems as though the locations aren't being applied until the app is restarted.
Any suggestions as to how this can be fixed? I'd really like to finish the fix to my app ASAP.
Thanks!

Shazron Abdullah
added a comment - 21/Mar/12 16:45 @daniel seems like it - more like a CB-330 case (re-opened)
@mike seems like a timing issue - in the instructions, try applying the code first thing in applicationDidFinishLaunching instead of webviewDidStartLoad. will try to fix
I'm away today - will get to your issues tomorrow.

Mike Newman
added a comment - 21/Mar/12 17:47 - edited Yep, I can confirm that it now works (at least for me) without restarting now that the database locations are changed during applicationDidFinishLaunching.
Thanks alot for the suggestion!

I think this implementation is still incomplete. After upgrading an app from <5.1 to 5.1, the OS will be 5.1 but the old location should be used. The check for location should allow the old. If the data exists, migrate it as well as update the plist entries.

Dave Orchard
added a comment - 22/Mar/12 04:33 I think this implementation is still incomplete. After upgrading an app from <5.1 to 5.1, the OS will be 5.1 but the old location should be used. The check for location should allow the old. If the data exists, migrate it as well as update the plist entries.

Shazron Abdullah
added a comment - 22/Mar/12 19:33 @Dave Orchard it is my understanding that in 5.1, the OS would move everything to the new location, is that not the case? Or is this only for new dbs in 5.1

This is a follow up to Bug ID# 11105407. After further investigation it has been determined that this is a known issue, which is currently being investigated by engineering. This issue has been filed in our bug database under the original Bug ID# 11063678. The original bug number being used to track this duplicate issue can be found in the Related Problem section of your bug report's Problem Detail view.

Shazron, was the bug id for the one you filed? They're taking a look at it.

Suhail Dutta
added a comment - 31/Mar/12 18:13 Got the following back from Apple
This is a follow up to Bug ID# 11105407. After further investigation it has been determined that this is a known issue, which is currently being investigated by engineering. This issue has been filed in our bug database under the original Bug ID# 11063678. The original bug number being used to track this duplicate issue can be found in the Related Problem section of your bug report's Problem Detail view.
Shazron, was the bug id for the one you filed? They're taking a look at it.

Shazron Abdullah
added a comment - 03/Apr/12 00:20 @Suhail no my bug id is rdar://11081309 but I know there are dupes at rdar://11081647 and rdar://10296507 as well.
So now we have these 5 dupes (at least):
rdar://11081309
rdar://11081647
rdar://10296507
rdar://11105407
rdar://11063678

I currently have Phonegap 1.6.1 and this bug is not fully resolved for me. Here's the flow:

1. Distribute the app wirelessly as an Enterprise app.
2. Data gets stored in localStorage
3. Distribute an update to the app wirelessly
4. The first time the user goes into the app, all of their data is gone. I use weinre and confirm that the localStorage shows up as empty.
5. The user must close the app completely, reopen the app and then the localStorage is back.

EDIT: If in step 4, a user starts putting in new data, it will get wiped out in step 5. Is there a different flow for the first time an app is updated because I would think it would copy over what's in the persistent storage back to the localStorage.

Roy Yang
added a comment - 24/Apr/12 19:57 - edited I currently have Phonegap 1.6.1 and this bug is not fully resolved for me. Here's the flow:
1. Distribute the app wirelessly as an Enterprise app.
2. Data gets stored in localStorage
3. Distribute an update to the app wirelessly
4. The first time the user goes into the app, all of their data is gone. I use weinre and confirm that the localStorage shows up as empty.
5. The user must close the app completely, reopen the app and then the localStorage is back.
EDIT: If in step 4, a user starts putting in new data, it will get wiped out in step 5. Is there a different flow for the first time an app is updated because I would think it would copy over what's in the persistent storage back to the localStorage.

@Roy, can you go to CB-330 and download and run the diagnostic plugin. How I did it is the only solution that I can think of - unless you have a *solution* to suggest you will have to run this plugin so I can get more clues to the problem.

Shazron Abdullah
added a comment - 24/Apr/12 20:32 @Roy, can you go to CB-330 and download and run the diagnostic plugin. How I did it is the only solution that I can think of - unless you have a * solution * to suggest you will have to run this plugin so I can get more clues to the problem.

Shazron Abdullah
added a comment - 24/Apr/12 21:47 Thanks @Roy I'll examine the log. You can make your users install the iPhone Configuration Utility http://support.apple.com/kb/DL1465 to access the console (select your tethered device, see screenshot: http://cl.ly/G6vW ).
I could update the plugin to paste the diagnostic log into the pasteboard as well if you'd like, so they can paste it into an e-mail.

Ah I see what the issue is. I looked at the console in xCode organizer. It appears to be the same error that @angelo has in CB330. Should I move my comments there to consolidate with that same issue?

It appears that the application gets a new ID or there is an inconsistency in the ID and I'm getting deny file-write-create /private/var/mobile/Applications/F2BE3CCA-AB36-4498-A7B0-81CB6D643512. On subsequent closing down and reopening of the app, I no longer see that message.

Below is after I wireless download an enterprise app. The first time the localStorage never initializes, need to close the app and reopen to get the right localStorage

The hypothesis is that the UIWebView grabs the location of the databases from the .plist, and from the report it shows that paths in the .plist are correct (path 15D8). At 16:50:47, it reports a security error when trying to write to path F2BE, which should never happen if the paths were read from the .plist.

The fix was to correct these .plist values if they were incorrect. In this case, 1. either the path was already correct, or 2. the path was fixed by the plugin. Was there anything before this log that showed a fix was applied at all, or is this the complete log?

Shazron Abdullah
added a comment - 24/Apr/12 22:26 This is quite strange, let me explain.
The hypothesis is that the UIWebView grabs the location of the databases from the .plist, and from the report it shows that paths in the .plist are correct (path 15D8). At 16:50:47, it reports a security error when trying to write to path F2BE, which should never happen if the paths were read from the .plist.
The fix was to correct these .plist values if they were incorrect. In this case, 1. either the path was already correct, or 2. the path was fixed by the plugin. Was there anything before this log that showed a fix was applied at all, or is this the complete log?

Shazron Abdullah
added a comment - 24/Apr/12 22:51 Thanks, this makes it clearer. The CB-347 fix was applied, and this is confirmed by printing out the .plist value, but UIWebView still picks up the old paths.
So either:
1. UIWebView is using another location for the .plist location and/or it's cached
2. The patch was applied too late
I'm leaning towards #2. I'm going to give the code a once-over and step through the timing of events.

There shouldn't be security errors before the page is loaded since there would be no calls to localStorage/WebSQL.

So the sequence of events appears to be correct, and they should occur synchronously. The only thing is, the UIWebView is created first before either of the code blocks, and perhaps during the UIWebView initialization, it picks up the .plist values.

In my testing, and probably in other users' testing when they tested the patch, this UIWebView initialization of .plist values happens after the patch is applied (a fluke of timing), so everything appears ok - while in your case, it is the opposite.

The evidence points to the UIWebView picking up the .plist locations after it is instantiated (which makes sense if one was to design the class), but before a request is handled.

The fix for this would be to apply the patch before the UIWebView is instantiated.

Shazron Abdullah
added a comment - 24/Apr/12 23:09 Looking at the code, the fix is applied here:
https://github.com/apache/incubator-cordova-ios/blob/master/CordovaLib/Classes/CDVViewController.m#L167
But the page is loaded here:
https://github.com/apache/incubator-cordova-ios/blob/master/CordovaLib/Classes/CDVViewController.m#L197
There shouldn't be security errors before the page is loaded since there would be no calls to localStorage/WebSQL.
So the sequence of events appears to be correct, and they should occur synchronously. The only thing is, the UIWebView is created first before either of the code blocks, and perhaps during the UIWebView initialization, it picks up the .plist values.
In my testing, and probably in other users' testing when they tested the patch, this UIWebView initialization of .plist values happens after the patch is applied (a fluke of timing), so everything appears ok - while in your case, it is the opposite.
The evidence points to the UIWebView picking up the .plist locations after it is instantiated (which makes sense if one was to design the class), but before a request is handled.
The fix for this would be to apply the patch before the UIWebView is instantiated.

@shazron Really appreciate your work.
This issue is blocking an update for our app, we see the same issues as described by @roy.
Do you see any way to avoid this issue using js or the phonegap API with 1.6.1 so we could build on phonegap build?

Urs Zimmermann
added a comment - 26/Apr/12 08:08 @shazron Really appreciate your work.
This issue is blocking an update for our app, we see the same issues as described by @roy.
Do you see any way to avoid this issue using js or the phonegap API with 1.6.1 so we could build on phonegap build?

@Urs I'm afraid that for PhoneGap Build there is no way you can patch it currently since all an uploader provides is HTML+JS+CSS assets. 1.7.0 release is slated for next week, and usually it takes a couple of days for Build to catch up. I'm sorry if that doesn't help.

Shazron Abdullah
added a comment - 26/Apr/12 21:00 @Urs I'm afraid that for PhoneGap Build there is no way you can patch it currently since all an uploader provides is HTML+JS+CSS assets. 1.7.0 release is slated for next week, and usually it takes a couple of days for Build to catch up. I'm sorry if that doesn't help.

So I just upgraded an app to Cordova-1.7.0 and tried to test data persistence during an iOS upgrade.

Since I have no iOS 5.0 device at hand, my idea was to launch the application in 5.0 Simulator, do some stuff which will be written to SQL/LocalStorage and then to copy the installed application via Finder to the 5.1 Simulator. Then I would proceed to launch the 5.1 Simulator (by running an empty cordova project from xCode), press the home button and launch the app I just copied.

Unfortunately, all the data is lost. Indeed, everything stored under Library/WebKit is deleted as soon as the application is launched.

Thus my question is: what is going to happen, when a user (with the cordova-1.7.0-based app installed) upgrades his iPhone from 5.0 to 5.1? Is the data saved by this fix (and only lost on the simulator) or will it be lost? And is there any way to test this without having to downgrade real devices?

alexander.koerschgen
added a comment - 03/May/12 23:49 So I just upgraded an app to Cordova-1.7.0 and tried to test data persistence during an iOS upgrade.
Since I have no iOS 5.0 device at hand, my idea was to launch the application in 5.0 Simulator, do some stuff which will be written to SQL/LocalStorage and then to copy the installed application via Finder to the 5.1 Simulator. Then I would proceed to launch the 5.1 Simulator (by running an empty cordova project from xCode), press the home button and launch the app I just copied.
Unfortunately, all the data is lost. Indeed, everything stored under Library/WebKit is deleted as soon as the application is launched.
Thus my question is: what is going to happen, when a user (with the cordova-1.7.0-based app installed) upgrades his iPhone from 5.0 to 5.1? Is the data saved by this fix (and only lost on the simulator) or will it be lost? And is there any way to test this without having to downgrade real devices?

@alexander this fix never deletes anything, certainly not the Library/WebKit folder. Did you verify on disk (for the Simulator) that it is actually deleted? If it is actually deleted, it's an Apple thing unfortunately.

Shazron Abdullah
added a comment - 03/May/12 23:53 @alexander this fix never deletes anything, certainly not the Library/WebKit folder. Did you verify on disk (for the Simulator) that it is actually deleted? If it is actually deleted, it's an Apple thing unfortunately.

However, I just went through my testing procedure once again. This time, I pressed the home button in 5.0 Simulator to force a backup. Then I copied the app to 5.1 Simulator again and fired it up. Again, the Library/Webkit was deleted but all data was still there – so I have to assume that the plugin found the Backup and restored the database.

This gives me confidence, that users having the new version of the app installed will not lose any data when upgrading from 5.0 to 5.1.

alexander.koerschgen
added a comment - 04/May/12 00:43 Yes, I did verify this.
However, I just went through my testing procedure once again. This time, I pressed the home button in 5.0 Simulator to force a backup. Then I copied the app to 5.1 Simulator again and fired it up. Again, the Library/Webkit was deleted but all data was still there – so I have to assume that the plugin found the Backup and restored the database.
This gives me confidence, that users having the new version of the app installed will not lose any data when upgrading from 5.0 to 5.1.
So thanks alot for your efforts!