Monday, 6 May 2013

Finally, some progress in video codecs.

ORBX.js, a downloadable HD codec written in JS and WebGL. The advantages are many. On the good-for-the-open-web side: no encumbered-format burden on web browsers, they are just IP-blind runtimes. Technical wins start with the ability to evolve and improve the codec over time, instead of taking ten years to specify and burn it into silicon.

I think the 'remote-screen viewing of videogames' use case is bogus (if anyone notices latency it's gamers), but this is a really important development for the reasons Brendan mentions and more.

I'd say video compression is maybe 2-4 times as efficient (in quality per bit) than it was in 1990 or so when MPEG was standardised, despite computing power and storage having improved a thousandfold since then.

Not much has changed. The video compression techniques we're using everywhere are direct descendents of 1980s signal processing. They treat video as a collection of small 2D blocks that move horizontally and vertically over time, and encode all video this way. If you want to make a codec work hard, you just need to rotate the camera. Partly this is because of the huge patent thicket around video encoding, mostly it's because compression gets less necessary over time as network capacity and storage increases. However, it was obvious 10 years ago that this was out-dated.

Meanwhile, there has been a revolution in video processing. It's been going on in video games, and in movies and TV. The beautiful photorealistic scenes you now see in video games are because they are textured 3D models rendered on the fly for you. Even the cut scenes work this way, though their encoding is often what compression researchers dismissively call a 'Graduate Student Algorithm' - hand-tweaking the models and textures to play back well within the constraints of the device. Most movies and TV has also been through 3d-modelling and rendering, from Pixar through visual effects to the mundane superimposition of yard lines on sports. The proportion of YouTube that is animation, machinima or videogame run-throughs with commentary keeps growing too.

Yet codecs remain blind to this. All this great 3d work is reduced to small 2D squares. A trimesh and texture codec seems an obvious innovation - even phones have GPUs in them now, and desktops have for 20 years. Web browsers have been capable of complex animations for ages too. Now they have the ability to decode bytestreams to WebGL in real time, we may finally get codecs that are up to the standards we expect from videogames, TV and movies, with the additional advantage of resolution independence. It's time for this change.

4 comments:

Maybe for playing games, but the market for watching other people play games is growing really fast. It's not just pre-recorded run-throughs on YouTube, live streaming is huge. Tournaments like Major League Gaming report peaks of 450,000 concurrent viewers, and even right now, early morning US time with no major tournament going on, there's 150,000 viewers watching mostly individuals stream their gameplay on twitch.tv, many of whom are their own little cottage industries, subsisting through subscription, sponsorship and ad revenue.

And the next generation X Box will natively support live-streaming.

If ever there was a time for a codec optimised for computer graphics, it would be now.

Maybe for playing games, but the market for watching other people play games is growing really fast. It's not just pre-recorded run-throughs on YouTube, live streaming is huge. Tournaments like Major League Gaming report peaks of 450,000 concurrent viewers, and even right now, early morning US time with no major tournament going on, there's 150,000 viewers watching mostly individuals stream their gameplay on twitch.tv, many of whom are their own little cottage industries, subsisting through subscription, sponsorship and ad revenue.

And the next generation X Box will natively support live-streaming.

If ever there was a time for a codec optimised for computer graphics, it would be now.

About Me

Kevin Marks works on IndieWeb and open web tech. From 2011 to 2013 he was VP of Open Cloud Standards at Salesforce. From 2009 to 2010 we was ay BT as VP of Web Services. From 2007 to 2009, he worked at Google on OpenSocial. From 2003 to 2007 he was Principal Engineer at Technorati responsible for the spiders that make sense of the web and track millions of blogs daily. He has been inventing and innovating for over 20 years in emerging technologies where people, media and computers meet. Before joining Technorati, Kevin spent 5 years in the QuickTime Engineering team at Apple, building video capture and live streaming into OS X. He was a founder of The Multimedia Corporation in the UK, where he served as Production Manager and Executive Producer, shipping million-selling products and winning International awards. He has a Masters degree in Physics from Cambridge University and is a BBC-qualified Video Engineer.One of the driving forces behind microformats.org he regularly speaks at Conferences and Symposia on emergent net technologies and their cultural impact.