Pilots: Too many ed tech innovations stuck in purgatory

Steve Kolowich wrote an article yesterday in the Chronicle that described the use of LectureTools, a student engagement and assessment application created by faculty member Perry Sampson at the University Michigan. These two paragraphs jumped out at me.

The professor has had some success getting his colleagues to try using LectureTools in large introductory courses. In the spring, the software was being used in about 40 classrooms at Michigan, he says.

Adoption elsewhere has been scattered. In 2012, Mr. Samson sold LectureTools to Echo360[1], an education-technology company, which has started marketing it to professors at other universities. The program is being used in at least one classroom at 1,100 institutions, according to Mr. Samson, who has kept his title of chief executive of LectureTools. But only 80 are using the software in 10 or more courses.

93% of LectureTools clients use the tool for less than 10 courses total, meaning that the vast majority of customers are running pilot projects almost two years after the company was acquired by a larger ed tech vendor.

We are not running out of ideas in the ed tech market – there are plenty of new products being introduced each year. What we are not seeing, however, are ed tech innovations that go beyond a few pilots in each school. Inside Higher Ed captured this sentiment when quoting a Gallup representative after the GSV+ASU EdInnovations conference this year:

“Every one of these companies has — at least most of them — some story of a school or a classroom or a student or whatever that they’ve made some kind of impact on, either a qualitative story or some real data on learning improvement,” Busteed said. “You would think that with hundreds of millions of dollars, maybe billions now, that’s been plowed into ed-tech investments … and all the years and all the efforts of all these companies to really move the needle, we ought to see some national-level movement in those indicators.”

In our consulting work Michael and I often help survey institutions to discover what technologies are being used within courses[2], and typically the only technologies that are used by a majority of faculty members or in a majority of courses are the following:

AV presentation in the classroom;

PowerPoint usage in the classroom (obviously connected with the projectors);

Learning Management Systems (LMS);

Digital content at lower level than a full textbook (through open Internet, library, publishers, other faculty, or OER); and

File sharing applications.

Despite the billions of dollars invested over the past several years, the vast majority of ed tech is used in only a small percentage of courses at most campuses.[3] Most ed tech applications or devices have failed to cross the barriers into mainstream adoption within an institution. This could be due to the technology not really addressing problems that faculty or students face, a lack of awareness and support for the technology, or even faculty or student resistance to the innovation. Whatever the barrier, the situation we see far too often is a breakdown in technology helping the majority of faculty or courses.

Diffusion of Innovations – Back to the basics

Everett Rogers wrote the book on the spread of innovations within an organization or cultural group in his book Diffusions of Innovations. Rogers’ work led to many concepts that we seem to take for granted, such as the S-curve of adoption:

Source: The Diffusion of Innovations, 5th ed, p. 11

leading to the categorization of adopters (innovators, early adopters, early majority, late majority, laggards), and the combined technology adoption curve.

Source: The Diffusion of Innovations, 5th ed., p. 281

But Rogers did not set out to describe the diffusion of innovations as an automatic process following a pre-defined path. The real origin of his work was trying to understand why some innovations end up spreading throughout a social group while others do not, somewhat independent of whether the innovation could be thought of as a “good idea”. From the first paragraph of the 5th edition:

Getting a new idea adopted, even when it has obvious advantages, is difficult. Many innovations require a lengthy period of many years from the time when they become available to the time when they are widely adopted. Therefore, a common problem for many individuals and organizations is how to speed up the rate of diffusion of an innovation.

Rogers defined diffusion as “a special type of communication in which the messages are about a new idea” (p. 6), and he focused much of the book on the Innovation-Decision Process. This gets to the key point that availability of a new idea is not enough; rather, diffusion is more dependent on the communication and decision-process about whether and how to adopt the new idea. This process is shown below (p. 170):

Source: The Diffusion of Innovations, 5th ed., p. 170

What we are seeing in ed tech in most cases, I would argue, is that for institutions the new ideas (applications, products, services) are stuck the Persuasion stage. There is knowledge and application amongst some early adopters in small-scale pilots, but majority of faculty members either have no knowledge of the pilot or are not persuaded that the idea is to their advantage, and there is little support or structure to get the organization at large (i.e. the majority of faculty for a traditional institution, or perhaps for central academic technology organization) to make a considered decision. It’s important to note that in many cases, the innovation should not be spread to the majority, either due to being a poor solution or even due to organizational dynamics based on how the innovation is introduced.

The Purgatory of Pilots

This stuck process ends up as an ed tech purgatory – with promises and potential of the heaven of full institutional adoption with meaningful results to follow, but also with the peril of either never getting out of purgatory or outright rejection over time.

Ed tech vendors can be too susceptible to being persuaded by simple adoption numbers such as 1,100 institutions or total number of end users (millions served), but meaningful adoption within an institution – actually affecting the majority of faculty or courses – is necessary in most cases before there can be any meaningful results beyond anecdotes or marketing stories. The reason for the extended purgatory is most often related to people issues and communications, and the ed tech market (and here I’m including vendors as well as campus support staff and faculty) has been very ineffective in dealing with real people at real institutions beyond the initial pilot audience.

Update: Add parenthetical in last sentence to clarify that I’m not just talking about vendors as key players in diffusion.

"Pilots: Too many ed tech innovations stuck in purgatory", 5 out of 5 based on 8 ratings.

I’m not arguing against faculty prerogative in technology adoption and for a centralized, mandatory approach, but noting the disconnect. [↩]

Related

Google+ Comments

About Phil Hill

Phil is a consultant and industry analyst covering the educational technology market primarily for higher education. He has written for e-Literate since Aug 2011. For a more complete biography, view his profile page.

The article started out discussing LectureTools, a specific application (i.e. product), and its low and stalling adoption rate (I hope that does not come off harsh to LectureTools, Echo360 or Mr. Sampson). I think the point Phil is trying to make though, is that we have all these great applications (again, products) that are not reaching a tipping point, critical mass, enterprise adoption… whatever, to help teaching and learning. I think this, in and of itself, is an interesting point: why are there so many individual efforts, many of which are apparently not garnering the level of recognition they deserve?

However then the article switches to a discussion of “technologies” (i.e. product categories, not products), pointing to several (AV, presentation software, LMS, digital content, file sharing applications) and tries to make the same point.

“Most ed tech applications or devices [product categories, not products] have
failed to cross the barriers into mainstream adoption within an institution. This
could be due to the technology [which, specific products or product categories?]
not really addressing problems that faculty or students face, a lack of awareness
and support for the technology [again, which?], or even faculty or student resistance
to the innovation [?]. Whatever the barrier, the situation we see far too often is a
breakdown in technology [products or product catagories?] helping the majority of
faculty or courses.”

I looked up LectureTools, which identifies itself as “Student Engagement and Assessment” and looks like it could fit into the product category of “audience response systems” e.g. clickers. (Again my apologies to all involved, I may have failed to recognize key differentiators between LectureTools and audience response systems. Please feel free to correct me with the proper product category if this is too far off.) So the question is not, “What is the adoption rate of LectureTools, and does that low rate indicate technology has failed campuses, courses, students and/or faculty?” but rather, “What is the adoption rate of audience response systems, and what does that high rate adoption rate mean to those groups?”

Otherwise, wouldn’t it would be fair to point to Sakai and, because it has one of the lowest adoption rates, infer that Learning Management Systems are stuck in pilot purgatory? I’ve found several lists of dozens of Audience Response Systems, and “site:.edu ‘audience response systems'” reveals 6,500 responses, while “site:.edu ‘classroom response systems'” yields another almost 8,000 responses. I don’t know how these Google returns map to campus use (i.e. adoption), but “Audience Response Systems” would appear to be on the downward slope of adoption at “early majority, late majority, laggards.”

Indeed it might be that for institutions the new ideas–a specific product, like LectureTools–are indeed stuck at the Persuasion stage, but at least in this case the particular product category–Audience Response Systems–is not.

Now after all that, I’m not arguing semantics–I’m arguing we have *too many starters and not enough joiners.* Or more generally, too many products in each product category. Why do we have innovative and creative faculty and campuses starting their own projects (products) like LectureTools in product categories (audience response systems) that are already crowded with viable options?

I would think the biggest factor in specific products not seeing increased adoption is that most edtech stakeholders are simply tired of the constant reinvestments required in the “adoption process” or rather, more formally on most campuses, the “procurement process.”

Thanks as aways e-literate / MindWires for a great discussion starter!

Patrick, you make some good points. My technique might have been sloppy, but the intent was to take the LectureTools example to talk about ed tech in general (mea culpa). Part of this interest of mine comes from recent conferences and seeing tons of examples of new companies and market hype (look what we could do) and almost no examples of results (look what we have done outside of anecdotes).

Where I disagree with your characterization is that I’m not implying that many deserve more recognition or success. More important to me is the notion that ‘something’s broken to have this many companies and this much investment without many cases of deep adoption’.

As for your suggestion of one big reason being too many products in each category – that makes sense in the LMS space or even audience response systems. But there are also cases of whole categories that seem stuck. Early alert systems, e-Portfolios, or adaptive learning systems, for example.

So there are at least two cases embedded here. Any thoughts on the categories that never get past pilots (for the most part)?

In so many cases, edtech firms do not think systematically about pilots. While getting a pilot is a milestone, it should not be a goal. The twin goals are widespread adoption in the institution (in K12 the school or district) and then leveraging the pilot to achieve adoption in a “critical mass” of institutions. The company needs to consider from the beginning:

1) how am I going to make the decision makers and users of the pilot outrageously happy?
2) how are we going to spread the pilot within the institution?
3) how are we going to utilize our happy customers to spread the word?

Of course, we at Academic Business Advisors would love to work with any organizations who want to do this better than they are currently doing.