Pushkit Performance Test

I decided to do a small test over time to see what the delivery times for a PushKit were. The results where both impressive and strange.

Here is the set up. Every 30 minutes I will send 32 push messages to two handsets. The first two messages are warm-up messages to get the connection to Apple servers up an running. I am just too cheap to keep my VM up and running at all times 😉

What did I measure?

Every push message included a timestamp when it was sent from the server. Since PushKit allows you to run code in background (thank you Apple), the app pushed that back to the web service, so in effect I got a total round-trip time for each push.

External circumstances

Since I didn’t measure only Apple push kit performance, due to the fact that I am doing round trip measurement, I want to disclose my setup.

I am hosting a web job on Azure and calling a web service to send pushes, and receive reports back from the handsets.

It’s a basic instance not always running in West of the USA. Regular network latency to this service for the phones is 150ms from the home wlan.

So when you look at the numbers below, 150ms seconds is for reporting back the numbers.

First glance test results

After running it for a day and 2370 push messages, here are the results

Average (ms)

Max time (ms)

Min Time (ms)

1672

61150

186

Nothing catastrophic: It’s not like a live open socket your iOS and the numbers seemed to be very scattered so I made a breakdown of average time of day. The most horrifying number is a max time of 61 seconds to deliver.

Now apart from a couple of spikes during the day, we seem to have a response time of less than a second except during the night. I suspect that my 2am–8am response times is Apple putting my phone on low power consumption because it learnt that I don’t use the phone that much during those hours. But it is of course Apple, so you just don’t know.

Let’s have a look at the two spikes at 11am and 8pm. Looking at the spike for 8PM I found the max value, 61150 occurred 20:51:36, and the message was to my phone. I live in an apartment building where the data connection is really poor and as it happens, I know I went out to meet Rinse for a pickup of laundry. Taking a closer look at all the pushes around that time, there are also 12 missing messages. So for that particular scenario I don’t think I can blame Apple, it is just my data connection.

Let’s take a look at 11AM. Same thing here, I was going out to the store and ended up with a bunch of 20+ seconds push messages as I was walking down the hallway.

Now I have a choice of simply removing the extremes, or maybe just look at my test phone. I want to test live usage so let’s keep them. But out of interest removing all over 20 seconds will give you an average turnaround time of 1138.

Distribution of response time

Let’s have a look at the distribution of response times. Now this looks more encouraging, almost 75% under 700ms and ~90% are under 3 seconds.

Trimmed results

For you that don’t want the numbers skewed by real life usage of a handset (such as not being online etc.), I took a look at 121 messages over 7 seconds, they are all lumped together and correspond to times I knew I was not in coverage.

Average

Max Time

Min Time

723

6756

186

As you can see, that improves the average a whole lot. But remember that this is not from a phone in someone’s pocket.

Lost messages

So far it has only lost 40 messages during the 24-hour test, and they seem to lump together while I was walking out of the building. Overall I think that you can trust that messages will be delivered. One worry is of course the timing and queue time Apple has. It seems to be quite a short time queued before it gets thrown away.

Conclusion

Pretty damn impressive in my opinion. This is only on their sandbox, and my backend is running on a basic Azure backend that is not always running. But for VoIP calling, getting the signal under 1 second 90% of time is pretty solid. A signalling time under 3 seconds for setting up a call is in my view acceptable. It’s just like the regular phone network.

What’s next?

I am going to let this test run for a couple of more days to see what it does to the numbers. It will be particularly interesting to see if I get more lost messages.

Comments

Related news

We’ve been busy working on some of our Android tutorials here at Sinch! In this blog post we’ll give you a quick run-down on what we’ve updated, and how each tutorial could benefit you and your customers. App to App… read more

Since 1957, when a five-year-old boy with perfect pitch first phreaked AT&T switches and invented phreaking, phones have been a target for different types of fraud that costs customers and phone companies billions of dollars. However, if you’re using the… read more

Verification serves as an effective method for securing your user base, reducing fraudulent or duplicate signups, and for two-factor authentication (requiring user to be in possession of the phone during signup). Sinch verification products can be integrated into your Android,… read more

We’ve given you some of the reasons verification is useful and becoming increasingly necessary, notably: Phone numbers serve as a username with longevity Reduce fraudulent or duplicate signups Two factor authentication (require user to be in possession of a phone… read more