Saturday, March 17, 2007

BBC Jam, £75 million for what?

Is the content worth the £75 million (yes read that again) that has been spent so far?

It was with a heavy heart that I tried to log in to BBC Jam, last time was a depressing experience. The whole experience was of an over-engineered project, where visuals and animation took priority over everthing else, including learning.

Loading….Loading…..Loading…..You get used to looking at ‘loading’ screens in BBC Jam. Huge amounts of time are spent waiting on something to happen. Then, as many things are introduced with an annoyingly elaborate flash animation, if you repeat something you get this stuff over and over again. This ‘animate everything’ approach disappeared from web design years ago.

Not another robot!An annoying floating robot (circa 1950), with one of those synthesised voices that remind you of cheap children;s TV, explains a settings toolbar without actually showing it on the screen. What is it with the BBC, robots and tinny voices? It’s the same in Bitesize – they’re everywhere. And why talk through a screen navigation sequence without showing the screen? In any case, the curiously dated robot crashed out with an error message before I could finish the session (just two minutes in). This happens a lot in BBC Jam.

Long linear tourI then took the tour. A linear animation with zero interaction and no user control. If you miss something, you’re lost. Even the simplest of video and animation delivery on the web has some rudimentary user control.

Design and technology The menu system on this module is a real hoot. I clicked on ‘food for though’ and got nothing but bouncing menu items and coming soon messages. Thoroughly confusing and a complete waste of time. The linear video and animation sequences were as dull as dishwater.

Fieldtrip for SEN (Special Educational Needs)The laboured animation at the start (you get used to this) has a dog that puts a VHS videotape (dated or what?) into a VCR and, you’ve guessed it, a video appears on a TV screen. By the way the back button doesn’t work. It has the wrong symbol (fast rewind) –should’ve been a vertical line and less than symbol to take you back to the start. Damn. an error message has appeared “A script in this movie is causing Adobe Flash Player 9.0 to run slowly” Not again. In a way I was glad as the content was dire. Largely a scrappy set of linear video and animation resources and horrific load times. I do the ‘game’. It’s a deadening experience wit pythonesque animation – again this is something that’s really common in BBC Jam – scrapbook animation. It’s tedious.

FrenchConfused from the start. They’ve improved the front-end menu since I last looked but it’s all hopelessly over-engineered. Basic design errors abound. For example an icon with a tick on it is the confirm button, yet the meaning seems to be ‘you got it right’. You have to press exit twice from each section, one would have sufficed. There’s also too much loading time, this was disruptive with endless countdowns and waits. Some just didn’t load at all, with no explanation.

First episode is a few cartoons – linear and next to zero learning. The second is video broken down into phrases, but some edit points are in the middle of words! Identifying the parts of the car was fine, although the vocabulary (windscreen wipers, licence plate, gears etc) was far too advanced for the age group. In another you have to identify words as you hear them, but this is just identifying what’s said, divorced from the meaning of what’s said. In some interactive exercises when you get things wrong there’s no formative feedback to tell you why or what the right answer is. The ‘make your own comic’ is fine, but is an exercise in sorting sentences and takes too long to navigate and complete. The DJ game is simply to identify masculine, feminine and plural, this is OK, but the vocabulary is far too complex at this stage.

The whole thing is VERY clunky and clumsy in navigation, style, interaction, vocabulary and learning. “I was fiddling around with it for ages and nothing happened. It was just a movie. It was crap. It’s confusing. I didn’t know what to do. I felt like it was, like, I should have been getting involved more as I was getting a bit bored. I thought it’d be better cause it’s BBC.” Carl (12 years old). Oh dear!

BusinessThe introductory menu is very strange. Loads of animation with buttons flying about, but when they settle, try clicking on the large orange button, nothing happens – classic design error – something looks clickable, but it’s not. You have to look for the barely visible, small rectangle below. Then I get the inevitable loading screen. This time it says the user-friendly word – ‘Processing…’. I do feel as though I’m being processed.

It gets weirder. One of its ‘star’ businesses is Eidos! I’d love to tell you about the other businesses, but hardly anything loaded and worked. What the BBC case study doesn’t tell you is that Eidos was days away from bankruptcy when this was being filmed. Their bank wanted to pull the plug and it was sold off cheaply after massive losses, missed deadlines and bug–ridden releases (a bit like BBC Jam). As an assignment you are asked to do a SWOT analysis of this now defunct company – that WOULD be interesting, if you had the real and current data to view!

The only interesting bit was the ability to explore the Eidos offices, but again, it was a lot of effort for very little reward. Why were all of the Eidos senior staff posing about for BBC film and photo-shoots at the very time the company was sinking - they should have been trying to get their lamentably late games out.

In general, the whole thing is a scrapyard mess. The repeated animations are annoying – same images over and over again – it makes you scream with rage. E-learning is about the user being in control. This is what you get when TV people create interactive content – thinly disguised broadcast material. Interactivity is the name of the game. Here you spend more time hanging about waiting, often on just simple pieces of repeated animation, than learning. Most of the time it’s like an animated PowerPoint in extreme and painful slow-motion. Try the Library – you may lose the will to live waiting on search results.

FinanceAt last, something that is really, really good. Don’t know who did the content, but it was well designed and the simulation approach to learning worked. It was way beyond the other content in terms of its focus on learning. It has a consistent interface, sadly lacking in the other content, good structure and a strong focus on learning by doing. Only a couple of niggles – figures should be lined up when presented in columns and some of the video sequences were overscripted.

Was the government & EC right to proceed with BBC Jam? NOWas the early content up to standard? NOWas the later content up to standard? NOT MUCHWas the content hopelessly behind schedule? YESIs the content produced so far worth £75 million? NODid the BBC break the government and EC rules? YES

20 Comments:

But you're not a 5-16 year old Don!! I don't know what your experience of the age group is, if you have any kids, have ever taught, have ever made teaching materials, but your tour of the content is conducted with seriously OLD eyes.

BBC Jam content was built and designed with the target audience from the word go. They were involved in the conception, paper-testing, alpha-testing, character design, voiceover casting... you name it, the kids were involved. And they love it! They really bloody love it.

Kids are much more screen-literate that we are, they snap things up in a second. Watching a five-year-old operating a laptop with a clickable trackpad is a sight to behold. They find the clickable things (by exploration, by instinct) they understand how to use menus, they come up with their own logic they don't need symbols. Their brains work in a completely different way to our jaded "its the wrong symbol, that button should be clickable, it should operate from left to right" concepts.

Knowing the content intimately, I'm going to make the following suggestions:

- French; you just didn't find all the content. It's very rich and very thorough. The kids never had any problems finding the hidden nuggets and treats. If you prefer conventional menus, you can always go into the modules through the Search and Browse curriculum menus.

- SEN Field Studies; it's designed for learners with severe learning difficulties and a mental age of around 3. The kids we worked with adore dogs putting videos into VCRs, they adore cartoony python animation. The concept of cause and effect is challenging at this level, so yeah - putting the video in the VCR makes the video come on, and a lesson is learnt.

- Finance: I made this, so, thanks. The kids that co-designed it with me in Wembley, Walthamstow, Bedfordshire and Leicestershire loved it too.

- Yes the infrastructure is poor, I've never been a fan of it, and those of us making the content were fairly united on that. One the EC conditions was that content should be portable into other VLEs via generic standards. This presented a number of challenges that I don't think BBC Jam met. It is slow, and clunky, doesn't do the amazing content justice.

- No, this is not £75m worth of content. Notwithstanding you have only mentioned half of the live content in your your, there is at least 3x more content than is currently live sitting on the shelves in White City queued up waiting to go live. It will now likely never see the light of day again thanks to the short-sighted misprioritised actions of the commercial companies that sought to prevent young people around the UK having unfettered access to engaging stimulating learning content, preferring to think about lining their and their share-holders pockets for the few years instead.

Thanks Sam. You are the only person from the BBC that has has put forward measured arguments under your own name. I really appreciate that.

I actually put my two 12 years olds through both the French and Business programmes, as I'm well aware of the 'old eyes' issue. I mentioned this in my post - quoting some of their comments. However I stand by my views and I'm not alone:

"From the beginning it smacked to me of adults imagining what children like, and seemed to have been produced with an eye on visual complexity rather than on effective on-line learning." Seb Scmoller (who has contributed wisely to this debate).

I have also spent 23 years designing, producing and delivering learning via computer technology, including many language programmes, as well as dozens of other educational topics.

That aside, I don't agree in designing menus that are difficult to use (that only the kids get). A menu is a means to an end , not the end in itself, many of the navigational structures are inconsistent and confusing (the finance programme being a notable exception).

I dod go through ALL of the French content, some of it twice as I reviewed it way back at the start of 2006. It was bad then and iot;s bad now. Read Graham Davies report (link via my post) - he's a lanuguage learnnge expert.

On the SEN introdcution, I still can't see why you'd subject a child with SEN to a sequence where a dog puts a VHS tape into a VCR machine to make a TV work within the screen of anoter machine. Some DVD children will have no idea what a VHS tape and VCR are. Why choose that dated and now almost disappeared example? It is not really tecahing causality if the objects which are causing things to happen are unrecognisable.

On finance I congratulate you and your team on this product. It was exemplary. It seemed to chime with the target audience and was a genuinely experiential learning experience. Most of the other content was not. I only wish the same level of design and consistency had been employed in the other programmes.

Why was the infrastructure poor. Portability to other VLEs, while not trivial, i a well trodden path. It doesn't seem like an adequate excuse for the crazy load times, error messages and general confusion.

Lastly, even if this only a quarter of the content, £75 milllion is way over the mark - by a golden mile. The project has not been pulled just because someone in the private sector has complained. It has clearly been mismanaged. There were conditions set on the BBC and they failed to meet those consitions, namely on budgets and consent.

You're welcome Donald (though I reiterate these are my views not the beebs nor my colleagues).

I can confirm, without doubt, it was not adults imagining what children like. The children told us and showed us what they liked. And that's several hundreds of children across the country, which I would assert is more representative than your two 12-year-olds.

And I disagree with you that, these days, a menu is just for getting to the content. The menu is a world, an environment, a door way. On my finance program, you talk with the characters, they introduce the content, they invite you to participate. Would you have preferred a white page with 4 links? Because the kids wouldn't have.

I did read Graham Davies critique of the French content, and I found it incredibly brief and particularly guilty of being done through "old eyes" - apart from using the galling phrases "loud throbbing music" and "spotty 14-year old males", he quotes decade-old academic texts. "Instructional software design" has moved on a lot in ten years.

I am as in the dark as the rest of the world about the exact reasons why the BBC chose to suspend BBC Jam (the EC have not indicated which conditions the service is said to have breached, so you there is nothing to indicate it is "namely on budgets and consent"). I don't work at Jam any more. But I will defend the content we produced and the people who produced it 'til I'm blue in the face :)

I work on jam. Well, I should say I "still" work on jam. As ever, the usual disclaimer that the opinions stated here are mine and certainly not those of the BBC as a whole. I haven't published my full name, but then I never do. Sam knows who I am so you could ask him for contact details if you like :-)

A couple of points of what I hope will be clarification. The total budget is £150m for the life cycle of jam. That includes absolutely everything - including the development costs of the user environment and back-end software infrastructure which is pretty complex I gather.

Content probably comes in at about £90-100m. There were going to be between 135-150 commissions - so clearly what's currently live (quick, it's gone tomorrow) is a drop in the ocean. To say that £75m has been spent on the subjects you have reviewed is (perhaps not intentionally) disingenuous.

For the record, £50m is guaranteed to go external in direct commissions. The rest goes on 'internal' commissions (people like me) - we sub-contract the design & build work. Both of these processes go through robust competitive bidding processes. So the majority of the budget is going to the commercial sector in a competitive way. Just not to companies like Pearson, Espresso, RM and so on.

I haven't experienced the problems you have in getting to the content so can't really comment on that but I understand that was being worked on. The original hissy fit from the commercial sector (or more accurately, a small but significant part of) created delays in the project timeline (a delayed start but fixed end) that meant that project infrastructure was being developed alongside content. It doesn't excuse problems with content but it perhaps explains them.

I happen to agree on the robot (as was) on the old intro screen :-)

The content was actually on schedule (in that the bbc promised to publish certain commissions in certain years and the only one where that happened was 2006. Up until the review began the beeb was on course, but agreed not to publish any further content once the review began)

Did the BBC break Govt and EC/EU rules? I don't really know how you can say 'YES' so unequivocally. As far as I'm aware a complaint has been made by BESA to the EU that jam is in breach of its conditions by not being distinctive enough from the commercial sector. This complaint has not been upheld. An EC review would probably take another couple of years to conclude whatever it would conclude. I understand people could infer some sort of admission of guilt because of the BBC Trust's actions, but perhaps rather than go through more lengthy, costly litigation it is easier and quicker to rescope the project. That's not insider knowledge but it's a possibility. Personally I'd have preferred a defence of the content as is, but you feel that there is more to come so we shall see. Que sera sera.

Having made some of the content (History 5-7 - take a look :-) I think it *is* distinctive. Not least because jam is actually made specifically for an un-mediated child, and not for teachers in whole-class activities.

Some commercial companies complain about jam being unfair competition because it's public money and not distinctive from the market products (which we could argue over til we're blue in the face!). Everyone has seen this coming. The market place should really have been preparing for this whilst growing fat on ELCs. That was pretty much guaranteed income for a couple of years. Surely that time could have been better spent to prepare for jam?

You say you've a long history in creating educational content and I have no reason to doubt that. But the wonderfully creative content producers (er, plus me) have a bit of experience between them too Donald. I have to say the blogs *do* read like you have a bit of an axe to grind, but that could be my inference rather than your implication :-)

I *do* know of many people who are deeply saddened by the loss of jam. Essentially the difference between us is that you think jam was a waste of money anyway and that the market can (or at least should) be able to fill the space. My position is that the market wasn't taking that opportunity and that it really was an innovative project that the users (children) loved - and they are the ones who are missing out.

Have to say 100% agree with Donald on this, the quality of the applications in the Jam setting is poor to say the least, the companies commissioned are also of varying quality and the products just look like the phrase "101 things to do with flash."

The answer to the "how do you get selected" question might answer some of these quality issues. I fear it has as much to do with have you ever worked for the BBC, know anyone who does or has worked for the BBC or do you have a big publisher name, than it does to do with quality of work. I speak as someone that has some first and second hand experience of the commissioning team.

There were/are plenty of innovative organisations out there who could have done tons better jobs with these examples than those developed. If a group of people who understand the audience, what e-learning is and isn't, the curriculum and its gaps, had selected 15 innovative organisations they could have spent 1/3 of this cash and delivered far better resources.

I speak as a parent, educationalist and developer and have to say good riddance!

As a link to your newest post Donald - this is more of a caterpillar project to me than a fruit fly, it was insular, inward looking, slow on the uptake, cocooned itself in a shell, came out with all colours blazing flapping its wings and died a quick death!

That phrase hit the bullseye dead centre. This post gets to the bottom of the matter. The excessive and inappropriate use of repetitive animation flies in the face of almost all web development and recent research in e-learning. The cognitive overload and wasted time on endless loading, animated menus and linear animated sequences reminded of all that dot.con stuff circa 2000. Did we learn nothing from Clark and Mayer, Nielsen, Norman and others on the overuse of fash animation in learning?

I too worked on jam. First as an internal staff member, then later on as a freelancer. I resigned from the BBC because I could not see how the Jam project I was working on was going to happen.

I did not see that the internal skillsbase was present (either locally or in London). I did not see any working methods that I recognised either from my days in web design, corporate e-learning or educational publishing. I couldn't understand why a 'commissioning' process was allocated up to half of a project schedule when nothing was built, tested or assessed during this point (ideas were just written down and talked about). A hierarchical organisational structure meant that an individual's skills and expertise were discounted over job title.

I personally did not see a lot of interaction with children - from what I know of different commissions, some projects sought children's opinions and created content with them, but others simply paid lip service to the idea that the child's opinion mattered

After I resigned from the BBC I worked on a number of very different e-learning projects across the UK and Ireland. But I was eventually tempted to go back to BBC Jam for the production of a subject that I care passionately about.

The old problems were still there - and unsolved. A lack of internal skills. A stifling and distant commissioning process. Trying to create content for a player that in your heart of hearts you knew wouldn't work. Trying to explain the concept of non-linear video and interactive content to people whose only background is TV and Radio. Creating content that you knew was going to be packaged up and held in a death grip inside the BBC Jam player - content that wasn't going to be used in mash-ups, distributed on mobile phones, taken by kids and used and reused. It was hard to work on BBC jam while watching Web 2.0 explode all around you. There were lots of new project titles floating around - project managers, technical project managers. And there were some new processes. But simple processes - like on-screen reviews of content to catch just basic spelling and factual mistakes - were not in place.

But we pushed through, and managed to create some genuinely interesting, innovative content for a niche subject that has NO interactive resources available online at all. And now this content is sitting on a server, still being tweaked and tested for a launch that will not happen. The concepts behind the content have not been shared. Our learning curve in creating non-linear reusuable video and graphical content has stalled, and our experience thrown in the bin.

BBC Jam was pulled because it was crap. There might've been some good content in there, but I didn't ever see it. Over-engineered, heavy flash. No freedom for the learner. A learning experience designed and suffocated by a team who are heavily into linear broadcast technologies. A project that thought it would succeed if only enough money was spent (a comment I often heard voiced in response to my concerns about the quality of jam was 'well, they've spent so much money on it now, they couldn't possibly take it away...').

Wikipedia - just text and graphics - lets learners learn and teachers teach on a scale never before seen. Wikipedia employs just five people. It's free.

The BBC need to conduct a post-mortem on the death of jam. To stop talking the talk about how great the content was and how sad it is that the nasty independents made it go away. They need to realise what went wrong and why. And then they need to take a look at their entire online model. Because the problems don't just end with Jam.

These are sane words from someone who knows what they're talking about. The observation that TV folk, talented as they are, don't adapt easily to interactive content is, I fear, right on the money. This, I suspect is the real root cause of BBC Jam's failure.

I too worked on Jam, across a wide range of commissions, and I haven't once dealt with someone from a TV background, every producer, developer I've had to deal with is from a new media background, so all these hack content TV producers, oddly, must have resided exclusively in northern Ireland.

I worked with Jam teams in Scotland and London as well, and met quite a few people from the BBC Wales. And while I met many people who had acquired online content skills (some being commercial imports, others having learned on the job in the BBC), I didn't meet many people who were specialised in e-learning, or who even felt it was a field in its own right.

I found that many of the content and editorial decisions in Jam (not the actual work undertaken) were made by people with a strong tv/radio background - naturally - the senior staff come from the more mature media - not 'new' media. Us junior online types always had to work around whatever decision was made.

I'd like to make it clear that at no point have I ever described Northern Irish online producers as 'hack'. As is usual to the BBC, the backgrounds and skillsets of the online department in BBC NI is varied and valuable.

I do not ascribe the failure of Jam to a particular set of people working on the project as a whole or specific projects within Jam.

I simply feel that the BBC's overall focus - linear radio and tv output - influenced Jam negatively. And this focus also negatively influences the BBC's online output overall.

I have met many talented and passionate BBC staff. The BBC is an amazing, frustrating, unique and demanding organisation. I wanted BBC Jam to succeed and could only watch in frustration as both the internal and external pressures stifled and finally suffocated the output.

I find all this bitterness and recrimination over the failure of Jam utterly negative. But an open and frank discussion of not only where outside pressures caused it to buckle, but how it was mismanaged from within, is the best way forward.

For what it's worth, I can hardly see how you have a problem with the Jam content. I come from a long e-learning development background and the Jam content was simply better than anything than was happening in private industry. It had bigger budgets, more ambition and more focus on user experience than anything I'd seen in the commercial sector. It was developed with constant user testing as good e-learning should be, and in general timelines were extended rather than sacrifice quality, something that also rarely happens in the commercial sector. Dont forget, the content was pulled because it was considered to hurt the competition, not because of poor quality.In my experience as a producer, you often get anecdotal criticism from individuals, sometimes they have agendas of their own. Your boss doesnt like the music, a teacher at a show doesnt like the colours, a parents twelve year old doesnt like it. You should analyse each crititism on its merits, and User test to see if this is shared by a majority of user. If you polled a bunch of teenagers you'd find quite a few that dont like Attenborough's Living Planet. So you should take Donald Clark's criticism in the same vein. Assess individually each point he's making, but without user testing to see how widely his opinions are share it's just another anecdote.

Even the most devoted Jam fan has problems with some of the content. The problems around the back-end infrastructure are well documented. Even the content producers admit this. The loading times are annoying and on the whole the linearity of the material and odd choice of illustrative content (Eidos) are all well documented. Even the CEO admitted to me at a conference that 'the early content was poor'.

You claim that Jam was"better than anything than was happening in private industry. It had bigger budgets, more ambition and more focus on user experience than anything I'd seen in the commercial sector"

OK, did you try learning basic French from the content on the site. The vocabulary was way beyond the comprehension of learners dealing with simple distinctions such as nmasculine, feminine and plural. It is, to be frank, not worth the effort. Over-engineered, Flash obsessed design, basic navigational design errors, that produced lots of linear animation with hardly any real learning.

True, it did have enormous budgets, more ambition and perhaps there was a focus on users (although some insiders from Jam who posted above think otherwise).

The sad truth is that for £75 million the actual output was low on usability, quality and quantity (with one notbale exception which I've reviewed elsewhere). I believe that it was pulled for a combination of reasons, one of which was poor product. However, the BBC also broke the rules that were agreed at the start of the project. Nobody, especially the BBC Trustees, pulls a £150 million project on the basis of a letter from a lobbying organisation. In this case they slit their own throats by failing to do what they promised to do - deliver outstanding content to a budget, against a set of rules agreed by everyone at the start. Sad but true.

For what it's worth I don't blame the developers or people who worked on the project. I blame the managers who are responsible for this huge cock-up - a disaster story that has been swept under the carpet (a great British tradition). £75 million down the drain - and no one really knows why. That's a true disgrace and not fair to either the Jam employees or people on the outside whose tax paid for the fiasco.

But the hidden tragedy in all this is that the established schools TV service that was run down to be replaced by Jam, will stop making any more programmes from midnight tonight. So at a stroke the BBC has killed off pretty well all its formal education for children. Step forward the commercial sector to fill the gap, as Espresso has just taken over Channel 4 back catalogue as part of their video on demand subscription service to schools. There's sure money in them thar classrooms!

There's money in the classroom alright -it's just that it won't be siphoned off into wht is regarded as one of the most costly and ineffiecient production houses around - the BBC.

We should be under no illusion that OUR tax money is far too often directed into the BBC or C4 to pay for the wages of expensive London-based staff. TV company learning organisations are not viable and soak up tons of money with negligible output. C4 is a good example. The C4 board have always see it as a pest and can't wait to get rid of it.

snipThe sad truth is that for £75 million the actual output was low on usability, quality and quantity/snip

75 million hasn't been spent on jam, only a fraction of that money - a previous post has referred to this, it might be worth reading what other people are saying that have more information than you. the very first comment posted in reply to yours will do for a start

furthermore, and again as stated by someone else, of this smaller amount that has been spent on content, only a fraction of it had been put live online pending a resolution on an earlier unsubstaniated claim by commercial sector companies who are a bit miffed at not being on the preferred suppliers list for jam (and who, i'm sure, can look forward to schools boycotting their inferior expensive products)

so please, it would be useful not to comment on 75 million pounds of content that you haven't seen much of

OK since you know how much was spent on BBC Jam, tell us. You can do this 'anonymously'.

Why is there so much secrecy surrounding the finances? Could it be that the BBC have failed to deliver against their promised budgets (a condition of them being allowed to proceed)?

As for the 'inferior expensive products' comment, I gave a fulll critique of the early content - it was garbage.

You sat"it would be useful not to comment on 75 million pounds of content that you haven't seen much of"

I haven't commented on unseen content. My critiques have been based on the released content, which was, on the whole, very poor. I have no reason to suspect that the unseen content was any better/worse.