TestFlight in iOS 8: Explained

Beta testing apps has long been a pain point for iOS developers. So it's no surprise that the announcement of TestFlight as part of iOS 8 was met with much fanfare at WWDC 2014. Since Apple's acquisition of Burstly (makers of TestFlight), there has been a lot of speculation and hope that Apple could finally release a more friendly solution for handling the distribution of beta apps. TestFlight marks a significant advancement for Apple in that area, and a welcome change for developers.

TestFlight vs. ad hoc distribution

Most people only ever install apps on their devices by way of the App Store. For people in the business of making apps, another method is frequently used: Ad Hoc distribution. Each iOS device has a unique device identifier (UDID). This UDID can be added to a developer account in order to provision the device for ad hoc distribution. This allows developers to distribute their apps for testing without making it publicly available for anybody to download. Managing ad hoc distribution requires developers to create and maintain provisioning profiles that specify what devices can run a particular app. This process is easy to screw up, can frequently lead to confusing errors, and most developers are limited to only 100 devices on their account. TestFlight seeks to change this.

The first significant change is TestFlight will not require developers or testers to deal with UDIDs or provisioning profiles. Currently, in order to add a new device, the flow goes like this: 1. Developer asks tester for UDID (and has to provide instructions on how to retrieve it if the tester doesn't know how) 2. Tester uses an application to retrieve the UDID 3. Tester sends UDID to developer 4. Developer logs into Apple's Developer Portal 5. Developer adds the tester's device to the account 6. Developer adds the new device to the appropriate provisioning profile 7. Developer updates app with new profile 8. Developer distributes app to tester

The exact flow may differ depending on what tools a developer is using, but that's more or less how it works. TestFlight's flow looks like it's going to be more like this: 1. Tester tells developer their Apple ID 2. Developer logs in to iTunes Connect 3. Developer sends email invitation to tester 4. Tester accepts invitation 5. Tester installs app via TestFlight app

If TestFlight can deliver on its promises, many of the frustrations of dealing with UDIDs and provisioning profiles could be a thing of the past.

1000 Apple IDs vs. 100 device IDs

The second big change addresses a long time complain of many developers — the 100 device limit. Developers will now be able to add the Apple IDs for up to 1,000 beta testers to their app. Though this comes with a caveat. TestFlight will require apps to go through a review by Apple. We don't know what guidelines apps will have to meet in order to be approved, and once an app has been approved, minor updates to the beta that don't significantly change the app won't need to be reviewed, but this is a new hoop for developers to have to jump through.

In addition to the 1,000 beta testers, developers will also be allowed to have up to 25 internal testers. Internal testers can't just be invited via email, they'll need to have an account created for them in the developer's iTunes Connect account. The advantage for internal testers is they won't have to wait for betas to be approved; they'll have access as soon as the developer uploads a new build.

After a build has been uploaded (and possibly approved), it will be valid for 30 days. If a developer goes more than 30 days without uploading a new build, testers won't be able to run the app until the developer uploads a new one. In addition to the binary upload itself, developers will also be required to enter metadata for the app. This includes an app description, as well as information about what testers should test.

Testers will be able to manage and install betas they've been invited to using the TestFlight app. TestFlight will only be available for iOS 8 when it's released, so developers still supporting (what will be) old iOS versions or Android won't be able to rely on TestFlight for those. The TestFlight app will allow users to view app descriptions, as well as testing notes. Testing notes will give developers a way to give their testers information about what needs to be looked at. Testers will also have the ability to submit feedback to developers from the TestFlight app (via email).

Latest version only

Another item worth noting here is it looks like all testers, whether beta or internal, will only be able to install the latest version of a beta available. In Apple's demonstration during their The New iTunes Connect session, the video shows all builds except for the latest being marked as "Inactive". When a new build goes up, the previously available build goes from having a checkmark to showing "Inactive" as well. Of course maybe developers will have the ability to control if testers get access to old builds, we can't say for sure until Apple documents it or we get access to the new iTunes Connect this fall, but this could be a deal breaker for many.

Crash reporting... later next year

One final big feature for TestFlight worth covering is crash reporting. When an app crashes on your device, a crash log is generated. iTunesConnect has long offered the ability to view those crash logs, but with limited success. One of the big missing pieces of functionality has always been lack of symbolication. Basically this means instead of a crash report telling a developer the name of the piece of code it crashed in, it would show the infinitely less useful hex address of that piece of code. Instead of something like "[OMGASIHTTPRequest reportFinished]", they'd see something like "0x9b000 + 23698". 3rd party services like HockeyApp have offered crash log symbolication for some time, and now iTunes Connect will finally have it. Unfortunately this feature will be coming "later next year", so developers interested in useful crash reporting in the mean time will need to stick with something else.

TestFlight in iOS 8: The bottom line

Ultimately TestFlight in iOS 8 means more options for developers and testers when it comes to beta testing. Developers will have the ability to distribute apps to more users outside of the App Store than they were able to before, and testers will get a sanctioned, native app for install 3rd party apps outside of the App Store for testing. And hopefully this expanded testing results in fewer bugs shipping to the App Store, and more polished apps getting into the hands of end users.

If you're a developer let me know — what do you think of the all new, all-Apple, currently all-iOS test Flight?

Overall, it was good to see the testflight product being rolled into the development tools suite. We will see how useful it is... If developers are limited to one active version, this will be a problem. I currently roll out versions with each build and testers often go between builds to validate changes and confirm issues. Additionally, if the beta build replaces the production version on the users device, I will have similar problems. Current beta users are my most active and important customers. If we may make a mistake in a beta they need to be able to use the previous builds.
There are some very big benefits from a test perspective. For one I will be using the same signing key in testing as release and releasing the binary that was tested. It would be great if we could add all the store content and add that to the test suite.

You're right. Depending on developers' needs and their current workflow, TestFlight may or may not be a good solution. I outlined some of TestFlight's restrictions that developers should consider here.

It could have been just fine, and nearly was, but Apple had to go and screw it up by introducing human App Store Reviews for Betas. Firstly, this introduces a long delay into what used to be an instantaneous part of the development process. Secondly, it makes TestFlight useless for Enterprise App developers: whose Apps are not necessarily produced with the App Store guidelines in mind. Apple just don't seem (or want) to get their heads around the fact that not all App Development is for the store.
After our first Beta rejection for a lazy misunderstanding on the reviewers part, we've moved to using TestFairy and haven't looked back.

I just went through an annoying problem with provisioning profiles this morning (had two copies of the same profile with different device id lists, Xcode kept using the wrong one), so this is welcome news indeed.

I hate Apple they ruined my career , with that shit , I m not able to test my app on my device not able to install the provisioning profile , with new IOS 8 , and finally they rejected my app after 20 days of being waiting , due the crash on iPad IOS 8 , that i don't know why , because it is working perfectly on previous IOS 7 . I'm lost