Multitasking Essentials for Media-Based Apps on iPad in iOS 9

iOS 9 on iPad introduces the ability to view and interact with more than one app at a time. Learn advanced techniques for efficiently sharing resources and presenting media while other apps are in the foreground. Tap into the potential of Picture in Picture to allow your video content to play above other apps, and explore multitasking best practices for creating great experiences in this environment.

WWDC 2015 - Session 211
- iOS

Resources

STEFAN HAFENEGER: Welcome to session 211,
Multitasking Essentials
for Media-Based Apps on iPad and iOS 9.
My name is Stefan Hafeneger and today Jonathan and I will talk
about adopting Picture in Picture
and mastering shared resources.
If you have already or plan to integrate a media-based app
on iPad and iOS 9 this is the right session for you.

This is the second in a series of three sessions
about multitasking on iPad this year.
By now you should be familiar with all
of the iPad multitasking features we have introduced
this week.
I want to remind you about the concept
about primary versus secondary app.
The primary app is always the one
on the left unless the device is set up right to left
which means it's the opposite.
In the first session we showed you how to get started
with multitasking on iPad in iOS 9.

In this session, I will show you how to adopt Picture in Picture.
When you are playing video on an application,
in iOS 8 when you are playing video, and you switch
to an application the video stops playing.
With Picture in Picture in iOS 9 the video automatically animates
into a floating window allowing you to use other apps
on the iPad while the video keeps playing.
I'm going to say Picture in Picture a lot in this section
so from now on I will refer to is as PiP.

PiP is supported on all of our latest iPads, iPad Air 2,
iPad Air, iPad mini 2 and 3.
Now, you might be wondering should my app adopt Picture
in Picture.
If your application provides an immersive video playing
experience, then absolutely.

If your application provides a kid's viewing experience
like a game, then probably not.
If you think your viewer was
like to keep videos playing while playing
with other applications.
Let's take a look at the video player APIs in iOS 8.

The media player framework has been
around since the first iOS SDK.
Last year we delivered AVKit
as a new high-level media framework, replacing parts
of the media framework.
AVFoundation is available for clients that want
to create their own video player with user interfaces.
[inaudible] using their technologies.
Let's talk about the media player for a moment.

In iOS 9 we are deprecating MP movie player controller
and MP movie player view controller.
In iOS 8 we are providing AVPlayerViewController
as a replacement for both classes.
And last year, in the Mastering Modern Media Playback session we
showed you how easy it is to adopt AVKit instead.

In this session we are focusing on AVKit,
AVFoundation and WebKit.
All three media frameworks rely on AVPlayerLayer
to display video content.
Let me show you how PiP works.
We start with an AVPlayerLayer in your application.

In order for the video to keep playing
when the application transitions to the background we need
to reroute the video frames to a different process.

For that, the system adds a new layer
that can host your video content.
When PiP is started, a PiP window is created in this layer
and positioned over an AVPlayerLayer.
Now, the video frames can be rerouted.
There is a seamless transition
and the video will just keep playing.
To be clear, we are not touching the AVPlayerLayer
on your application.

We just stopped delivering video frames.
Now, assistant can position
and size the PiP window however needed,
and the video currently keeps playing even
when your application transitions to the background.
So which of our modern video players support PiP?
I'm happy to announce all of them.
In the first part of the session I will show you how to adopt PiP
in your applications using all three media frameworks
and we will start with AVKit.
AVKit's main API for video playback
is AVPlayerViewController.

A modern AV controller subclass
that provides standard controls and behaviors.
In iOS 9 AV view controller smarts Picture in Picture
and it's a button in the lower bar.
When the user taps this button the video automatically animates
into the PiP window.

My colleague Felix will show you now what exactly you have to do
in your AVPlayerController application
for the PiP button to show up.

FELIX HEIDRICH: Hello, everyone,
I'm excited to show you the new Picture in Picture support
of AVPlayerViewController.
For this I have a demo application running
in the simulator that shows a collection
of videos using a collection view controller.
I can tap on one of the videos
to reveal detailed information about it.

I can also tap the big play button to bring
up AVPlayerViewController and start video playback
in full screen, but as you can see, the user interface does not
yet contain a PiP button in the lower right corner.
For the button to appear, two simple steps are required.
Let me show them to you in Xcode.

The first step is to set the apps background modes.
For this, I switch to the project settings,
select the targets, and go to the capabilities tab.

Here under background notes need to be enabled and the audio
and air play check box needs to be selected.
The second step is to set the app's audio category.

For this I go to the app delegate
and first import AVFoundation.
Next, an application did finish launching this option.

I can set the audio session category
by asking AV audio session for its shared instance
and then setting the category
to AV audio session category playback.
With this code in place, let's launch the demo again.
I again select the video, start playback in full screen,
and now in the lower right corner,
the PiP button is present.
When I tap the button, the video starts to play
in Picture in Picture.
The user interface in the PiP window contains three buttons,
the central button toggles video playback and I will talk more
about the other two buttons in a bit.
Also notice that there is a playback progress indicator
present in the UI.

Now, while the video is playing, I can navigate back
to the collection of video, of videos and, for example,
look at detailed information of a different video.

I can also simulate pressing the home button
and launch a different app all while the video continues
to play.

On the left button in the PiP window should return video
playback to the application.
When I tap this button, the demo app is brought back
to the foreground, but as you just saw,
the PiP window is simply dismissed and that's
because our PiP machinery does not know if the app is ready
to handle video playback.
Let me show new Xcode how to restore the user interface
to handle video playback just as you're users will expect it.

For this I switch to the collection view controller.
New in iOS 9, we are adding a delegate property
to AVPlayerViewController
and in this code I already made the collection view controller
the delegate of AVPlayerViewController.
To reinstall the user interface the collection view controller
needs to implement a simple delegate message has a very
descriptive name and is called player view controller,
restore user interface for Picture in Picture stop
with completion handler.
In this code, I ask the navigation controller
for its top view controller, and then use the top view controller
to present the player view controller
that the delegate provided.
The other thing in the code is
to call the completion handler once the player view controller
is presented on screen.
By doing this our PiP machinery know
that it can start animating the video back into place.
So let's run the demo again.
I again tap on one of the videos, play it in full screen,
switch to Picture in Picture and now when I return video playback
to the app,
AVPlayerViewController is presented
and the video nicely animates back into place.
Another thing I want to show you is what happens
when I leave the application while the video is playing
in full screen.
For example, if I now simulate pressing the home button,
in this case, we automatically start Picture in Picture
so that the video continues to play.
At any time users may tap the right close button
to dismiss the PiP window and that's Picture
in Picture support of AVPlayerViewController.
Thank you.
STEFAN HAFENEGER: So let's review what you just saw.
You have to do two things.
First, you have to enable the correct background mode
otherwise your application won't be able to keep playing
when it transitions to the background.
You do this in Xcode [inaudible] capabilities.

Audio and air play is the one you need to enable.
In the future iOS 9 [inaudible] rename to audio play
and Picture in Picture.

Second, you need to set a valid AV audio session category.
A good place to do this is did finish launching with options.
You ask for the shared audio session,
and then set the category to AV audio session category playback.
If the application also supports audio recording,
you can alternatively use AV audio session play and record.

If the application supports background audio
on air play iOS 8 you are already doing both things.
If you think the application should not support Picture
in Picture in iOS 9 we have a new option
on AVPlayerViewController that allows you to disable it.
Notice how Picture in Picture is automatically started
when the user taps on the message notification.
Why does this happen?
When the primary application transitions to the background,
an AVPlayerViewerController is presented full screen,
the system starts Picture in Picture on the user's behalf.
If the video [inaudible] is playing, then Picture
in Picture is possible.
Keep in mind though that the user can turn off this behavior
in settings in general multitasking persistent
video overlay.
Felix showed in the demo
that you automatically dismiss the AVPlayerViewController
and the user starts Picture in Picture.
We don't know the structure of the application,
or where the user navigated to when it was active,
so we need your help to restore the AV viewer controller
by implementing this delegate method.
In a simple case you could just call presentViewController
animated, by using the [inaudible].
A few applications have more sophisticated structure
than the demo application, you might have
to do a few more things.
In either case, don't forget to call the completionHandler
when you are done with the restore.

And that's AVKit and AVPlayerViewController.
Next let me show you how to do Picture
in Picture using AVFoundation.

As a client of AVFoundation you have an AVPlayerLayer
and most likely player controls.
I'm going to show you what you have to do and get from here
to here in your applications.
In iOS 9 we have a new class called AV Picture
in Picture controller.

It allows you to implement same PiP viewers I just showed you,
but using your own user interface.
Before I create an instance of an AV Picture
in Picture controller you should check whether Picture
in Picture is supported on the current device.
You create an AV Picture in Picture controller
by providing an AVPlayerLayer.
Think of AV Picture in Picture controller as an objects
that allows to present the content
of the AVPlayerLayer in the PiP window.
Setting a delegate is optional but it is very likely
that you will need it.

Next, you have the PiP button to use interface.
So that the user can start Picture in Picture.
You should only add the button
if the device supports Picture in Picture though.
There are situations where starting Picture
in Picture is currently not possible,
even though the device supports it in general.
In this case, you should disable your PiP button.
For that we have a property on the PiP controller
that you can observe and then update the enable set
of your PiP button.
The implementation of your PiP button action will look
something like this.
You first make sure that PiP is not already active.
And then you can start Picture in Picture in the pipController.

Do not call this method without user interaction.
If you use this inappropriately the App Store team will
reject submissions.

If the application provides additional content while PiP is
active you may want to dismiss the video player viewer
controller after PiP is started.

In order to do so could implement the did start delegate
method and then call dismissViewControllerAnimated
on your controller.

When you do this though make sure
that you don't release your PiP controller.
Because if it does happen, the PiP window will disappear.

So make sure you structure your application accordingly.
AV Picture in Picture controller has a delegate method
for restoring your view controller.

In a simple case you could just call present view
controller animated.
Again, make sure you call the completionHandler
when you are done with the restore.
Because when you call the completionHandler it is assumed
that the AVPlayerLayer used
to initialize the PiP controller is back on screen.
If this is not the case, we won't animate
to video frame spec. Your AVPlayerLayer won't show
on video frames while PiP is active
so if the application provides a non-modal view player experience
like this example you might want to update the user interface.

AV player viewer controller shows [inaudible] like this
and hides viewer controls.
This behavior is not required
but we highly recommend this for consistency.
In order to do this you could implement the start delegate
method, hide your player controls
and show your place holder art work.
When PiP stops you do the opposite,
you implement the did stop delegate method,
hide your placeholder art work
and show your player controls again.
Very simple.

Finally if you want to allow PiP to automatically start
when application transitions to background you have to be sure
that the AVPlayerLayer covers the entire UIWindow.

If this is the case, and the video is playing,
and PiP is currently possible,
the system automatically starts Picture in Picture
on your user's behalf.
And you will see all of this in action,
we have brought a great sample project you should check
out after the session.
So you saw how easy it is to adopt Picture
in Picture using AVFoundation as well.

Finally, let me show you Picture in Picture with WebKit.
WebKit's main API for video play is WKWebView.
This part of the session is for those among you using WebKit
to present your [inaudible] web technologies.
In iOS 9 WKWebView supports Picture in Picture
for HTML5 video if the application is set
up the same way as for the other APIs.
If your web content uses iOS default
in the playback controls, PiP will just work.

If you are a website developer interesting
in incorporating a PiP button into your controls check
out section 501 what's new
for web developers in WebKit and Safari.
If you want to support background audio air play
but don't want to allow Picture in Picture,
we'll have a property in WKWebViewConfiguration
in the future iOS 9 seed that allows you to do so.
And that's WebKit.

So you saw how easy and straight forward it is to adopt Picture
in Picture in your applications for all three media frameworks.
I can't wait to see PiP support in all
of your iPad video player applications.
Picture in Picture is a lot like background audio.
If the application is not on screen
but the user is enjoying your media.
The same rules apply to background audio
and air play apply here as well.

For instance, do not perform any task unrelated to playback
and limit your memory usage to only these tasks
when the application is in the background.

Properly sharing resources is critical not only for PiP apps,
but for all iPad applications.
And to discuss this further let me welcome Jonathan on stage
to tell you everything you need to know about this topic.
Thank you.
JONATHAN BENNETT: Thanks Stefan multitasking
on an iPad allows you to use multiple apps onscreen at once.
But hardware resources are still shared between all
of the presented apps.
I would like to explore with you some of the iOS media policies
and updated best practices you can use to manage audio, video,
and camera in your applications with multitasking on iPad.
The good news is that some
of these best practices should already be familiar.

And many apply to all iOS devices so it's easy
to make your app shine everywhere.
Now, before I dive into specific media resources,
I'd like to take a few moments to discuss some
of the roles your app may take on iPad.
In iOS 8 apps are presented full screen.

And new in iOS 9 you can have up to three apps on screen at once
with the new multitasking modes.
These apps may be vying for some of the shared resources
so to help manage that, the system categorizes your app
into one of three roles.
Full screen apps are considered primary.

And they continue to be so when a secondary app is presented
in slide-over.
The full screen app also remains primary when it's resized
to fit next to a pinned app in split view.
And a Picture in Picture application can be put
into the background while the video remains visible
to the user.
In this case your app's considered background media.
For certain shared resources these roles help define the
system's policies and your app's capabilities.
So let's dig into some of these resources.
Starting with audio.

The audio system on iOS is well-suited
for multiple applications.
If your app currently using audio,
you have configured the AV audio session API to state the nature
of the audio in your application.
The system uses this to figure
out how your app's audio mixes with,
docks or interrupts other app audio on the system.
The good news is a properly configured AV audio session
should just work with multitasking on iPad.
To this end, never change your audio configuration based
on how your app is presented on screen.

It's all about the audio.
With that, I would like to go over some
of the audio session best practices most relevant
to multitasking on iPad.
If you are not following these practices,
your app may have worked okay on iOS 8
but showed issues on iOS 9.
First off, only activate your audio session
when audio is first needed.

This is especially true
if you have an application whose audio interrupts other apps
such as music or video application using audio sessions
playback category.
Never solely determine to activate your audio just
because your app launched or came into the foreground.

Wait until the user interacts
with your audio feature before activating your session.
For example, a music and video application would wait
until the user taps the play button.
Next, there is a huge category of apps that don't need
to interrupt other audio.

For example, if you have a game or have other ancillary audio
like sound effects, you should use the ambient audio category,
this category provides appropriate behaviors
for this kind of audio including mixing with other applications.
This is critical for these kind of applications on,
with multitasking on iPad, because your app may remain
in the foreground while other apps come and go.
And you don't want your game audio interrupted
and having the user have to restart your app in order
to get its audio back.
Lastly, some apps have secondary audio Soundtracks.
An example of this is if you have a game
with bold sound effects and a music Soundtrack
that plays during the game.
Now, if the user is already listening to music
in another application,
you would like to silence your secondary Soundtrack.
So that the user it continue listening
to their music while your sound effects still mix in.
We provided a great way in iOS 8 to know when to do this.
Simply check AV audio sessions secondary Audio Should Be
Silenced Hint and listen to its began and ended notifications.
So these are some of the most important best practices
for using audio on iPad,
and as a bonus following these will insure a great experience
on all iOS devices.
If you would like to learn more
about configuring your audio session, there is a bunch
of great tips in last year's Core Audio talk
and in our detailed programming guide.

Next up, video.
And first off, a lot of video has audio associated with it.
So if you have an app that provides movies or TV shows,
you will want to properly configure your AV audio session.
In that example, you will want
to set audio session's category to play back.

Next. If you support air playing video in the background
or add the new PiP support or add the new PiP feature
in your application, you will be considered a background media
app when you are playing media in the background.
This means that you will have lower maximum memory limits
so it's really important that you discard any unnecessary data
such as view controllers, views, images, and other data caches
that aren't necessary when your app is off screen.
You will want to do this proactively.

Don't wait for a memory warning to do this.
You will also want to limit yourself
to performing just the task necessary to perform playback
in order to help share the CPU.
Now, many iOS video apps provide their content via HTTP
live streaming.

If you have a streaming video application,
you will want to do two things.
First, make sure that you are providing multiple variants
in your stream including a smaller resolution,
low bandwidth variant.
And second, make sure you are annotating all of your variants
with the resolution attribute in the master play list.
Now, if you get your video from an external vendor,
be sure to check with them to see if they are providing you
with these diverse well-annotated variants.
And this is important for three reasons.
On iPad and iOS 9 your video may not take up the entire screen.

Providing well annotated variants allows iOS
to choose the right-sized video
for how it's currently presented.

And when it's presented in Picture
in Picture a smaller resolution low bandwidth variant lowers
your memory footprint, and helps avoid terminating apps due
to memory pressure.
And lastly, following these practices help users avoid
unnecessary data plan usage.

And they will really appreciate that.
The last topic I would like to talk about today is camera.
In iOS 9 only one app can use the camera at a time.

In addition, for certain camera features they may only be
available when one app is visible on screen.
This means that camera availability can change
at any time and your app's camera usage could be
interrupted because the user is interacting
with other app using multitasking.

For this reason, it's important
to consider how people use your application
and its camera features.

If you have a camera centric application, you may want
to consider being full screen only.
Our camera app is a great example of this.

If your users expect an experience like this,
you can add the UI requires full screen key to your info P list.
This will help let users get quick access
to the camera inside your application
and use the entire iPad as a view finder.
Now, if using the camera is not the primary feature
of your application, you will likely want
to adopt the multitasking enhancements on iPad.
Our notes app is a great example of this.

People are going to love the flexibility
of using the new notes features inside
of slide-over and split-view modes.

However, this mean that's some
of the app's camera features may not always be available.
So let's take a look at how our camera
and notes apps handle camera availability on iPad
and how your apps can do the same.
When one app is on the screen,
UI image picker can let you see a preview, take a photo,
or do a video capture, just like today.
And when multiple apps are on screen,
you can still see a preview and take a photo, however,
video capture will be disabled.
So let's dig into this case.

Here we have full screen maps
with your app presented in slide-over.
The UI image picker is in photo mode so you can see a preview
and the user can take a photo if they tap the shutter button.
When the user swipes over to the video mode,
they will see a message that says your app needs to be
in full screen in order to take video.
What does this mean for your application?
First off, active video captures can be interrupted at any time.

An example of this is if you are recording video full screen
and an app slides over a new app in the slide-over feature.
Your video will be interrupted and it's important to note
that this kind of interruption can happen
to any video capture app regardless
if you support the multitasking modes or full screen only.

Next, the UI image pickers start video capture method may fail
because we are in a multitasking mode that doesn't support it.
If you are not currently paying attention to this return value,
now is the time to do so.
So these are some of the things for UI image picker.
Let's transition to AV capture session now
and see how the API differs.
When one app is on the screen,
you can still have full access to the camera.

And because AV capture session is a flexible low level API,
you can do more than just these three features listed here.
And we have been amazed at some of the creative
and powerful features you have added in your applications.
In order to ensure that we can offer you the resources
necessary to provide the highest quality camera experience inside
your application, an AV capture session using the camera will
only be available when one app is visible on screen.
This means that whenever slide-over, split view,
or Picture in Picture is active,
your camera's session will be interrupted.
You should handle these interruptions and communicate
to your users that the camera is currently unavailable.
Let's see how you can do this.
The first step is to listen
to AV capture sessions interruption reason
or excuse me, was interrupted notification.
And new in iOS 9,
the notification's user info dictionary states a reason
stating why your camera was interrupted.
There is multiple reasons you could be interrupted
but when it's due to multitasking
on iPad you will be told that the video device is unavailable
with multiple foreground apps.

This is your opportunity to adjust your UI
by disabling any capture buttons and also identifying
that the camera is currently unavailable.

Now that you have handled these interruptions,
what do you do when they end?
When an interruption ends,
AV capture session automatically resumes the session,
there is no need to manually restart it.
You can be notified of this
by observing the interruption ended notification.
And, again, this is your opportunity
to restore your camera UI by re-enabling any of the buttons
that you previously disabled.
Be sure to check out our updated AV cam sample code for examples
on using these new interruption reasons inside your application.

So in review, Picture in Picture is a great new feature
for video playback applications.
You can adopt it easily with any
of our modern playback frameworks and remember
that if you are using MediaPlayer framework
for video playback to transition over to AVKit to get the benefit
of PiP and other great features.
Also be sure to follow best practices Stefan talked
about to ensure a great experience inside
your application.
As an example, make sure to restore your UI
so that the video animates right back into your app.

Multitasking on iPad provides the power
to use multiple applications at once.
However, this comes
with a responsibility of sharing resources.
Three things you can do are
to properly configure your audio session,
provide adaptive video streams
and handle the new camera interruption reasons.
For more information, get in touch with our evangelist
and check out the forum.
And also we provided a new document called adopting
multitasking enhancements on iPad.

Now, sharing media resources is only part
of the story for shared resources.
Be sure to check out the optimizing your app
for multitasking session happening next up in Presidio.
There is also a great Friday talk
on delivering performance on iOS.

And if you missed it,
the getting started session is a great place to learn
about adopting adaptive UI in your apps.

On behalf of Stefan, Felix and myself.
It's been great to talk to you this afternoon.
Thanks, and have a great rest of the week!

Looking for something specific? Enter a topic above and jump straight to the good stuff.