Wednesday, February 4, 2015

This is the second in the series comparing sample test applications using vendor native development technologies (Objective-C, Java), Cordova (using Microsoft's Multi-Device Hybrid Apps), classic Xamarin and Xamarin Forms. This time I decided to focus on IO using the different frameworks. For this I used SqLite to save and retrieve records and also just writing and reading from plain text files.

The Test Apps
Applications were made for each of the development platforms that are functionally similar. There was little (if no) effort to make them look exactly the same or even look "good". But they looked about the same.

The Timing Methodology
Due to difficulties in knowing when things are "done", particularly with JavaScript, timings were handled via stopwatch. Each timing was taken ten times and the results were averaged. It should noted that hand timings have an accuracy of about 2/10 of a second so that does give us an approximate margin of error. In the previous post I showed the 10 individual timings and the average. This time to save me some typing I'm just showing the average of the ten timings.

Test 1: Test App Size
As mentioned in the last set of test, the size of the application can impact how much bandwidth it takes to deploy and also have some impact on load times. For Android the size of the APKs was examined. For iOS I looked ipa files for ad-hoc deployment.

Development Platform

Size

Android

Java

921kb

Cordova

417kb

Classic Xamarin

2.1mb

Xamarin.Forms

3.4mb

iOS

Objective-C (64 bit)

55kb

Cordova

621kb

Classic Xamarin

2.95mb

Xamarin.Forms

8.17mb

A few interesting things to look at here. First is the Cordova apk is actually smaller than the vendor native tools on Android. I suspect this has to do with differences in size of the SqLite libraries for both of the platforms. Also while the Xamarin APKs are larger, they are not as large as they were on the first set of tests. For this I used the linking option of Link All Assemblies. This was probably important for Xamarin Forms which is, of course, just a library on top on Classic Xamarin. For whatever reason I was not able to get the Link All Assemblies option to stay with Xamarin iOS and you see the results with Link SDK assemblies only instead.

Test 2: Load Times
Like the last test I verified how long the applications took to load into memory. The results were similar to the set of tests I did previously.

Development Platform

Test Avg.

Android

Java

1.044

Cordova

4.068

Classic Xamarin

1.689

Xamarin.Forms

3.032

iOS

Objective-C

1.14

Cordova

2.138

Classic Xamarin

1.204

Xamarin.Forms

1.928

As last time vendor native technologies load the fastest. Xamarin Classic follows closely with Xamarin Forms somewhat after that. In all cases, Cordova is the slowest loading. It is interesting that on iOS Xamarin Forms loaded almost as slowly as the Cordova app. Of course for iOS, nothing really took that long to load.

Test 3: Adding 1,000 records to SqLite
I originally wanted to add 10,000 records but found that too slow on my Android device and also it didn't scroll well under Cordova where my implementation had no auto paging. Offline storage is a common need with mobile applications and SqLite is one of the few solutions that span all platforms.

* I have had a few comments that there are better performing ways to do this test. For example I could have inserted 1,000 records in the same transaction or if I wanted to do it one one record in one transaction at a time, I could have done it asynchronously. Both of these points are undoubtedly true. Through SqLite you can get 1,000 records into the local database in much quicker ways on all platforms than this tests indicates. What this test is doing is looking at synchronously saving a single record in a single implicit transaction and doing that 1,000 times for all platforms. It would probably not be too difficult to do the other tests as suggested and they may be the source of a future post. Thank you to everyone who commented.

Right off the bat you might notice that Xamarin on Android did poorly. Very poorly. Slower than native Android or Cordova by a large margin. One thing I noticed is that in native Android I used the SQLiteOpenHelper but this is only available in the Android API. I used the method I did because it was cross platform and you can see the results on Android. SQLiteOpenHelper is available in the SqLite Xamarin Library for Android and on a hunch I tried it instead and you can see those timings under Xamarin Classic Alternate for Android and the timing were virtually the same as they were for Java. I suspect if I used this for Xamarin.Forms I would have gotten about the same advantages. The lesson from this is just because there is a cross platform version of the API, it doesn't mean it will perform well. Buyer beware.

Test 4: Querying SqLite Records
I wanted to test reading the 1,000 records I just wrote. I only did the 1,000 I originally wrote because I didn't want to deal with paging solutions for Cordova. A 10,000 record test performed very poorly when just creating a 10,000 for HTML table.

Reading records in Android was similarly bad for Xamarin on Android until I switched to the SQLiteOpenHelper implementation and then performance came online with the native example. As usual the interpreted JavaScript in Cordova slower but probably won't be a factor unless there is a large operation reading thousands of lines.

Test 5: Writing Lines to a File
I also wanted to test writing lines to a text file and how that performs. For each platform I write 1,000 lines to a file.

When it comes to writing lines to a text file the vendor native technologies are the undisputed leaders in high performance. Xamain on iOS is close and a little slower on Android. Cordova comes in at 3-4 times slower than the other technologies. Some of this may come from the interpreted looping logic and not the plugin but the reality is that the overhead of the interpreted code that calls into the plug happens if you write one line or a thousand.

Test 6: Reading Lines from a File
Compliment to test 5, reading those 1,000 lines from a file and displaying them on a list.

You want fast, go for native. Xamarin is as good or nearly as good except for Form on Android. This may have more to do with the performance of the list display than the actual reading of the list. That should be comparable to classic Xamarin. Same comments on Cordova as before, it's slower. If this is a factor or not depends a lot about the type of application.

I hope some of you find these comparisons helpful. The lessons I learned from this set of tests were:
- It is self-evident that not all implementations of code even on the same development platform perform the same. Having said that, be careful of wrappers around libraries that have unified APIs to allow for cross platform code. It may not perform as well as the specialized libraries for Android or iOS.
- While it is hard to differentiate slowness in code that comes from it being interpreted and what comes from the libraries they are using, to some extent it doesn't matter. In production apps the Cordova JavaScript code will be interpreted as it calls into plugins like SqLite so both will be a factor.
- The linking settings on Xamarin can have a huge difference in the size of the app. If you are using external libraries where you are not likely to use all the functionality use the Link All Assemblies option if possible.