Many Forks in a New Road: The UX Hell of Cross-Platform Content Viewing

When iOS stopped supporting Adobe Flash back in 2010, significant cross-platform audio and video rendering issues surfaced that remain with us today.

Article No :1073 | August 19, 2013 | by Pete Mackie

In the past, webbrowser page content like text and streaming video or audio always worked, spare for occasional rendering issues.

It didn't matter if we viewed web content on a Mac or PC, using a preferred browser like Internet Explorer (Windows), Safari (Mac), Firefox, Chrome, or Opera.

Text content followed HTML4 layout specifications, while video and audio streaming was played back using the Adobe Flash player, which was preinstalled on the browsers.

Mobile devices like smart phones and tablets changed all that. Initially, viewing mobile web content, whether static text or streaming video, worked just fine. No one had to give UX a second thought as Adobe supported a mobile version of the Adobe Flash player for all mobile web browsers' platforms.

Then, starting three-plus years ago, viewing identical web content on any computing platform became the expected norm. We expected that content viewing on our desktop computer would render just as well on our smart phone or tablet. Computing platform users observed that viewing content on any device frequently created unacceptable UX.

Those in the computing trade or closely affiliated stated: “No problem, we are moving from content rendering software using Adobe Flash and HTML4 to HTML5 with a new <video> HTML tag to replace Adobe Flash."

The software industry assured computing users, "We’ll have a short transition period while websites cease using Adobe Flash while simultaneously moving from HTML4 to HTML5 content layout inside of the web browsers. Soon your inconsistent UX will go away.”

Others continued to scratch their heads wondering why the promised consistent content viewing on any computing device was not always there. After all, ubiquitous platform content viewing was promised three years ago. The promised delivery of good UX on any computer platform is still not there and will not be for another year, if not longer.

This issue has led to a considerable amount of frustration, as users do not always know why they can’t see sites in the same way they can on their desktop and laptop computers and mobile devices.

Smartphone penetration is already a reality. “Two separate surveys confirmed that smartphone penetration has not only passed half of all mobile subscribers, but has gone well beyond 50% of all adult Americans for the first time," Forbes reported back in June. "The Pew Research Center places the figure at 56%, up from just 35% two years ago while noting the number of adults with no cell phone at all has fallen by half in that time, to just 9%.”

The emerging mobile platform—replacing PCs with tablets and smartphones—is driving the need for more care when it comes to rendering all forms of HTML content across all browsers on all platforms. There’s more to the story, though. Actually, the software development community unwittingly and unnecessarily foisted this inconsistent content viewing problem on users starting back in April of 2010.

How and Why it Happened

Lets look back to January 2010, when cross-platform viewing of rich—not just plain text—web browser and email HTML tagged content was not an issue. The web browser HTML layout control tags—a.k.a. HTML4—rendered cross-platform text and graphic images just fine. Sure, there were cross-platform rendering and view problems to overcome, with a little extra work for video, audio, and web animation we used Adobe Flash, which was (fortunately) ubiquitously deployed in all web browsers on almost all computing platforms, both PC workstation and mobile devices. Flash videos rendered the same way anywhere without much in the way of video and audio differences.

It was a time when Adobe Flash won for online rendering of video, not because it was the most ideal method, but because a developer could publish a video using the Flash video codec and expect it to be viewable most anywhere. Write once and view anywhere was the norm. Then, suddenly, in April 2010, all hell broke loose when our cross-platform rich content rending was unequivalently forced to take a new platform road for displaying rich content on mobile devices.

The big bomb: Apple, principally Steve Jobs, announced that their mobile device operating system, iOS, would no longer support the running of Adobe Flash on the iOS devices—iPod, iPhone, and the soon to be released iPad. Apple’s reasons for no longer supporting Flash on the iOS operating system were covered in Steve Jobs now famous online post: Thoughts on Flash.

Video display via the <video> HTML5 tag element using the H.264 (also known as MPEG-4) codec

Adobe continued to support Flash on Google’s Android smartphones that were, and still are, available from several vendors. Then, in November, Adobe’s support for Flash was pulled on all mobile platforms. Almost overnight the entire software development community started to abandon the use of HTML4 web layout with Flash as the rich content display for user viewing. In its place came web browser content display using: HTML5, JavaScript, and CSS3, and video display using the HTML5 <video> tag element rendering the H.264 video codec.

The dying off of Flash did not leave clear and concise alternatives. This is exactly when and where the UX hell was created that is still with us today. We were moving down a new road, with multiple forks in front of us:

Use of HTML5 web layout instead HTML4, where the HTML5 standard was basically an idea, far from being a standard specification for consistent multivendor use

Not all web browser providers were planning to support H.264 as the rich content video (and audio) codec

Not all web browser providers were ready to move from HTML4 to HTML5

Some readers who have made it this far are probably saying: “The solution is easy, just stop using content encoded in Adobe Flash.” But the issue is much more complicated than not using Flash encoded content. These are the forks awaiting us along the new road content developers and publishers are forced to take.

Fork: Nonexistent HTML5 Web Specification

The first fork appeared back in April of 2012: the lack of an HTML5 specification to develop website content against. But wait a minute, isn’t the web browser the ubiquitous standard for rendering content across all platforms? That is what we are told, but web browsers are far from being consistent and reliable at rendering content.

The published W3C HTML5 specification is neither complete nor finalized. Working with an ever-shifting HTML5 specification is part of a whole phenomenon, which creates a mix of available web browsers with different content rendering functionality. Your text and video content may or may or may not render to the user properly, depending on the web browser and its respective version being used.

Fork: Adobe Flash to HTML5 <video> Element

This video (and audio) fork in the road is more involved that you might realize.

Missing is an HTML5 <video> tag codec standard. The idea is that you feed streaming video to an HTML5 <video> tag and it will render properly for the user. This is yet to be determined within the HTML5 specification as to which video codec or codecs to support.

Presently each browser provider has a different idea regarding which video codec to support. You can see what I mean by viewing the UX Magazine article, The State of HTML5 Video. In the Media Formats chart, observe that different browsers support varied video codecs. Confusing to say the least.

What this means is that your video is viewable in one browser but potentially not viewable in another. To me, it is unconscionable that the software industry is even allowing this to happen.

One solution is to use adaptive codec publishing (create multiple codec formats) and send a different video codec file depending on the web browser being viewed. Another alternative for video publishing on your website is YouTube. You can optionally embed your YouTube videos on your website. There are tradeoffs with this approach beyond the scope of this article.

So where does the HTML5 <video> codec battle go from here? When will these video codec standards be established? There are no answers yet and we will not have them for another year at least.

Fork: Web Platform Streaming Video on All Computing Platforms

Web publishing just for Windows and Mac OS X workstations is declining and being offset by the explosion of disparate mobile devices. Associated with this is the rapid emergence of BYOD (Bring You Own Device) to work. These are typically tablet devices and sometimes capable smart phones.

We’ve lived through three-plus years when all web browser providers where each browser supports a different set of video codecs. What this means is a given video will render fine in one browser and not even start up on another web browser. This problem is exacerbated when viewing videos between desktop and mobile devices.

An employee may view video web content on their home desktop. Then take their tablet or smartphone to work only to learn the video is not viewable at work. The same scenario can happen in reverse too, from work desktop to mobile BYOD video viewing. The non-technical computing platform user, of course, does not have a clue as to why this is happening and for that matter “why does this not work?”

The cause of this platform inconstant video rendering problem is most obvious and straightforward. For the W3C HTML5, yet to be released specification, <video> HTML tag there is not a list of supported and unsupported video codecs.

Indeed this is a very bad situation: various web browser providers—principally Apple, Microsoft, and Google—are all vying for their favorite video (and audio) codec to be supported. Today, browser providers are like a bunch of Wild West cowboys. “We have a larger video content installed base than browser provider X, therefore our HTML5 <video> HTML tag should be the W3C supported standard.

As I mentioned, this non-ubiquitous platform-to-platform video rendering problem will be with is for another year or more. First, the final HTML5 specification will not be released until the last quarter of 2014. Second, there will be three-plus years of old published web content in the wild, which will require republishing to the final HTML5 specification.

Content updating will not happen immediately. Video rendering consistency will not happen until the specification W3C HTML5 <video> HTML tag supported list of codecs is finalized and published. There are software development paths around this issue using specially crafted HTML5 <video> tags or the use of what is called video adaptive bitrate encoding, but that’s another story for a subsequent article.

Conclusion

Inconstant web content rendering, especially from the desktop to the BYOD smart phones and tablets has been with us now for three years. Expect another year plus of these issues and realize that, despite all your UX design and management, sometimes your content will not render as you desire—especially on BYOD devices—and that this issue is beyond your control for a while longer.

Comments

Cenk Aksu

December 21, 2013

As a content management system administrator for my company I agree. The User Experience Platform (UXP) seems to be the platform that will tighten the ranks. Content management systems (CMS) like Microsoft SharePoint are just too decentralized. I’ve dealt with this problem for years. It’s hard to find a CMS that fits the needs of a business and user with portal similar conditions, social capacities, email broadcasting, and a centralized database making corporate data easy to find.
I’ve only enjoyed one system with this kind of User Experience, Centralpoint by Oxycon. Centralpoint gives us the comfort we need heading in to the new tends of technology.

I read up on this Centralpoint content management system and for some reason I’m not getting the same thing as you, Jen. It looks like a pretty complicate system. We are a small company with content management needs. My company has a little less than 30 something employees and about 10 of us play very important roles and work in an environment where a high level quality management is conducted for clients. The CMS that we use fit our needs just fine and I really can’t see how and company of are size could benefit from Centralpoint. Could you give me some ideas of Centralpoint being used on this level?

re: we are indeed, left out in the cold to hand-code replacement solutions back in the world of massive browser incompatibilities.

I could not of described this issue any more crisp and distinct than you above statement. Until you, as a developer, attempt to provide a consistent UX experience across disparate browsers on multiple platforms, using HTML5, JavaScript, and CSS3, you have no idea on how nearly impossible this development assignment can be. One has to know a lot more than knowing how to define and develop computer code.

The Fall of Flash was indeed a terrible blow for web UX reliability. Worry free cross-browser support and the ability to use vector assets for core elements were two of my main reasons for using it in many cases. Since there is still no 1 to 1 replacement for Flash, nor are there any tools I know of that can automatically convert Flash projects into html5—we are indeed, left out in the cold to hand-code replacement solutions back in the world of massive browser incompatibilities.

The article still reads: "...Write once and view anywhere was the norm. Then, suddenly, in April 2010, all hell broke loose when our cross-platform rich content rending was unequivalently forced to take a new platform road for displaying rich content on mobile devices.

The big bomb: Apple, principally Steve Jobs, announced that their mobile device operating system, iOS, would no longer support the running of Adobe Flash on the iOS devices—iPod, iPhone, and the soon to be released iPad. Apple’s reasons..."

Umm... no, that's wrong. By 2010, the train had long since left the station. The iPhone had been out - without Flash - for nearly 3 years at that point, so basically every content provider had already made the switch. Steve Jobs' "Thoughts on Flash" letter was simply a confirmation of why Apple would *continue* to not allow Flash; there was no change in policy.

And as for "snatch[ing] defeat from the jaws of victory": Mobile Safari commands about 80% of mobile browser share. Just exactly how much more would it take for you to consider it "victory"?!?

I almost entirely disagree with your basic premise. This branching didn't occur when (as Shaun correctly points out) iOS /announced/ they would not support Flash ever, but long before the iPhone was launched. Mobile devices have been forcing odd content on us for almost a decade. And by no means did web video always work. In the era you long for many organizations restricted or banned plugins, so video was essentially impossible to load for a large percentage of the users.

Settings aside video, it's a much larger issue. And by issue I do not mean "problem." It's an opportunity, and one that is pretty easy to deal with if you've been designing properly (not "colors and shapes" but APIs and flows sort of designing) all along.

Many have. Two years ago is when I first heard guys from NPR discuss how they had, for years already, been preparing content for everything from RADIO to the desktop web. With plenty of mobile web, mobile apps, tablets and oh yeah, radio visual display summaries. Check out the history of electronic programming guides; yeah, people all the way back in the 1970s who realized that the world would change and their stupid paper ones would not last forever, so set up flexible databases whose structures we still have.

Check out rates of information consumption over *SMS* sometime. Forget HTML5, or a specific video standard to export to, how about all these other channels? It's not a one time conversion, but a way of life.

So…your first "fork" here is converting from HTML4 to HTML5…which I'd argue doesn't make a ton of difference. Especially if you're talking about problems in rendering on mobile devices, none of which run IE, making life really a lot easier.

Then the next two forks are basically the same thing, how videos on the web are a pain if you're not using Adobe Flash. Which is definitely true, but I don't exactly see the difference between your points?

You've got a lot of the details wrong here, specifically "March 2011 [..] Steve Jobs announced that their mobile device operating system, iOS, would no longer support the running of Adobe Flash on the iOS devices"

iOS devices have *never* supported Flash. Why do you say it's only been two years?!

No, you haven't. The article still erroneously implies that iOS once supported Flash and that it was later removed. That is patently false, as iOS has never supported Flash. Jobs' "Thoughts on Flash" letter was only an explanation of why, and why that wasn't going to change.

My article research notes clearly indicate, to me, that the Steve Jobs: Thoughts on Flash article was published in April 2010. Somehow, and I'm not exactly sure why or how how, this Steve missive date got translated to March 2011 throughout my article. Probably my date and time duration was entered once and then carried throughout. It was an inadvertent error and not lack of research, though this should have never happened. I apologize for missing this rather gross oversight on my part. I do truly appreciate you pointing this oversight out to me.

re: iOS devices have *never* supported Flash.

Above, I think you mean Flash Mobile as a released product on iOS. I would agree with your statement, Flash Mobile was never an iOS released product. Steve Jobs April 29, 2010, "Thoughts on Flash" online post says no to Flash on iOS using the word Flash 35 times in his article post. Apple snatched defeat from the jaws of victory. Steve's online post statement, "The avalanche of media outlets offering their content for Apple’s mobile devices demonstrates that Flash is no longer necessary to watch video or consume any kind of web content," allowed me in this article to make the statement. "The big bomb: Apple, principally Steve Jobs, announced that their mobile device operating system, iOS, would no longer support the running of Adobe Flash on the iOS devices..."

The idea of write once, deploy across many platforms/devices is patently wrong when considering touch devices. The simplest explanation is "hover", a predominate Flash feature, is broken on every touch device. The list goes on.

Also, as a mobile developer, Google's support of Flash, taking the high road, as if Apple was wrong in the harsh judgement was a political move. What i'm saying is that as Adobe cried foul, Google put their arms around Adobe and said its not that bad, we love you, we allow flash on Android. This garnered praise for Google and Adobe, standing up for the mobile users rights. Google knew it was a terrible experience, that mobile CPU's could not render Flash effectively (at that pint in time), at the very least to be a pleasing user experience and would soon die anyway being a proponent of HTML 5. Google won by standing with Adobe, bashed Apple and never lost face in the transition.

Being that the focus of UXmag is about the users experience, I find it disturbing that this article does not recognize the pathetic nature of Flash on mobile/tablet devices at every level. Heck, Android code could not access the GPU on most mobile handsets at the time of this cry for justice, much less would the same handsets receive an OS upgrade from the handset mfg. at any time near Google's code releases. So much more than meets the eye here. Fortunately most of the 2.2 code based handsets have died so developers can really forget about them and not have to support that code base.

If Flash truly had a chance on Mobile/Tablets, Adobe would have continued to support it, as CPU and cores increased in the space and Google was behind them (cough).

SO, in my opinion, this point or argument for Flash is moot at best.

Thank you for letting me share my opinion and love the majority of content from this site and the contributors. I learn a lot and its helped my career immensely, especially Mr. Mackie's contributions. =]

I do agree that "the idea of write once, deploy across many platforms/devices is patently wrong when considering touch devices." I do observe though that several website software developer tool providers are making the claim of write once, run everywhere. Has this "write once" notion been truly proven yet? I don't think so. Lots to watch for here in what shakes out—if anything—and which multi-platform development tools will slowly fade away. The solution, at least for now, is observe and be cognizant of what may or may not work in the way of website software development tools in the way of form, fit, and function.

re: I find it disturbing that this article does not recognize the pathetic nature of Flash on mobile/tablet devices at every level.

I particularly do appreciate this comment. The thrust of my article was that Adobe Flash was disappearing on mobile devices and replaced with a incomplete HTML5 specification. I would have like to same more regarding "the pathetic nature of Flash on mobile/tablet devices," but the UX Magazine editorial guide lines strictly limits the size of Contributor's articles.

On mobile, we are past Adobe Flash and for some mobile applications to applying the the use of HTML5 specification. Next should be to more discussion about what HTML5 does provide in in the UX way of features and benefits, or lack thereof. For example, just applying use the HTML5 is worthy of one UX Magazine article is worthwhile. I will write this article if I sense that there is UX centric interest in this topic.

The rapid pace and plethora of options, we can all use help to head in the right direction. Community and comments help me so much to see whats new and fresh outside my local coffee shop circle of dev friends.