Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

NOW TV’s customer base has tripled year on year, and linear event streaming has become a greater and greater percentage of their viewing. This talk will take you through the lessons we learnt following one unfortunate event in 2014 which led NOW TV to catastrophic failure, and the improvements and scalability challenges we were faced with when preparing for the same event just 1 year later.

Hear how we turned our entire failure case behaviour on its head; from always failing in favour of NOW TV, to failing in favour of the customer. Learn how we ensure that high availability of video assets on the CDN ensures continued playout irrespective of the state of our application servers. Take away some tips of how NOT to scale, as well as how we reversed our fortunes and delivered stability during the biggest event on NOW TV to date.

No cover-ups or side-stepping our previous faults, hear about the rough with the smooth, along with the customer sentiment that has made us a far better service today. Get an insight into NOW TV’s platform and architecture from CDNs, through our Java/Groovy/Scala application servers, down to our Mongo databases, and learn how we will handle the load expected when we double our customer base in the coming year.

2.
2
• Tom Maule
– Solution Architect at NOW TV, Sky
– Previously Senior Java Developer on NOW TV Platform team (since project inception in early 2012)
I have also previously worked in the defence and telecoms industries
tom.maule@sky.uk
linkedin.com/in/tommaule
@tommaule
Who am I?

4.
4
Introduction - Overview
• NOW TV is the online no-contract TV streaming service from Sky
• Available on over 60 devices including the award-winning NOW TV Box
• NOW TV oﬀers movies and entertainment VOD and linear content, and for the first time in the
UK, pay-as-you-go Sports linear content

5.
5
Introduction - Customer Base
• Our customer base TRIPLED in the year up to April 2014
2013 2014

10.
10
Linear Streaming
• Unlike other OTT (Over-the-Top) Providers, NOW TV oﬀers streaming of live channels
• This is typically NOT predictable
• Load is driven by live events, not by time of day
Linear VOD

20.
20
What happened?
• High load stressed our database
• Retries only compounded the problem
• Observed issues:
– Customers couldn’t start new streams
– Existing streams were terminated
– Concurrency errors during and shortly after the outage
– Very high read and write queues in Mongo DB
– Entitlement and Viewing History APIs performed very slowly
– High proportion of time was spent updating indexes in Mongo DB

28.
28
Heartbeating: Future
• Game of Thrones Linear customers produce ripple-eﬀect heartbeating
– Due to heartbeats fixed to a 10 minute period
• In future, we will randomise the first heartbeat period in attempt to smooth out these ripples
H C E P V I M

44.
44
Entitlements: Today
• Entitlement checking by tag(s) was introduced
– /entitlement/tags/movies
• Entitlement checking now only needed to occur once per collection or ‘section’ of the app
• Where entitlement checking by content ID is still necessary
– tags are cached in memory
E
P V I MH C

59.
59
Recognition
MongoDB Innovation Award 2015
recognises organisations that are
creating ground-breaking
applications. These projects
represent the best and most
innovative work in the industry over
the last year.
DTG Innovation Award 2015
recognises organisations
which have driven innovation
in a particular technology or
sector