What would technology do best for learning?

An evolving set of best practices would help educational technology projects

Given the opportunity to use technology to improve learning in a school or in the community, what would you do? What technology would you use? What would you develop? Whom would you serve? There are all kinds of answers.

Recently, as part of a team judging submissions that proposed new ways of using technology in education, I had a good chance to see a variety of proposals from many different teams, providing a variety of answers to those questions. However, I also found myself thinking how many of the proposed projects could be better. I wished the teams themselves were a better judge of their own proposals, and that they understood how their project advanced appropriate uses of technology in education.

I wished that each of the applicants had been able to consult an evolving set of best practices for developing educational technology projects. Such guidelines might help educational developers understand and share learning about technology stacks and encourage development based on open standards. They might even define standards of quality and methods of interaction and engagement. They might help others avoid pitfalls and learn from failures.

The question is not what technology would do for learning. Instead, the question we should be asking is how could more of us collaborate to develop projects that demonstrate innovative uses of technology in education? If we are going to see real benefits from applying technology in education, we need education to become an open platform for innovation that leverages the work of many people working independently.

Based on my experience as a judge, and a limited experience at that, I’d like to share some thoughts on creating projects and improving project proposals, which might lead toward developing a set of best practices for open educational technology projects.

I don’t mean to suggest that I’ve understood or captured best practices here, but you have to start somewhere. I invite others to revise or extend this draft.

Here’s a list of observations about project proposals, as seen from the eyes of a judge in a competition. The ability to create a good proposal should relate to the ability to build and manage a project, just as angel investors make decisions about which startups to fund based on pitches and presentations.

Don’t do what others are already doing.

I suspected that many proposals were repeating something that had already been done, sometimes with little variation. Worse, they didn’t realize that it had been done before. Or they simply believed that it was worth having them do what others were doing.

Do something new and explain what’s really new about it.

Proposals should supply a set of references to related work that’s already been done. Knowing that related work exists is a selling point; ignoring it is not.

Explain how you’ve found a new opportunity that others have not identified.

Know why it matters.

The biggest question anyone should ask about any proposal is why does it matter? The answer should be the mission that brings together the team and informs all the decisions. Moreover, in a project proposal, the answer should be really obvious. It starts with the basic description of what the project is.

You should be able to describe your project in terms of the practical benefit it will deliver.

Who benefits from the project?

How do they benefit?

Is the benefit lasting and will it extend beyond the period of funding?

Set expectations clearly and don’t over-promise.

Make technology choices that are realistic and well-informed.

With some projects, I could not tell why certain technologies were chosen. I wondered if developers had previous experience using the technology. Did they have realistic expectations for what a technology could do? I also wondered how thoroughly the team explored other options. In other words, most proposals lacked a good rationale for selecting the technology involved in the project.

Describe the team’s previous experience with technology and whether you are familiar with its strengths and weaknesses.

Following “technology trends” seldom makes a proposal worthwhile.

Limit the scope of new development.

Some proposals were technology “kitchen sinks.” The team was promising to do everything because they don’t know how much work is involved in doing it. For example, a project proposal mentioned creating a social network to support their application. Will they use an existing social network like Facebook or will they create their own private social network?

Create a clear but constrained definition of the project

Build on an existing platform to limit the risk of taking on too much.

Integrate with technology that exists instead of building your own.

Focus on defining the core technology that must be built.

Develop the team doing the work.

Judges in a competition don’t get to interview the team face-to-face. We end up having to trust reputation often in terms of traditional credentials or affiliation. Or we use Google to find out more about the team and its members. Mostly, we have to make guesses about the experience and expertise of the team. If one viewed the project strictly as a business investment, we’d want to know as much about the team as we could because the team matters so much to the success of the project. The idea or vision of a project usually takes a backseat to the people driving the project. We’d want to know that the team had a shared vision, specific responsibilities and that we could trust them to deliver.

Who is the leader?

Who does most of the work?

Who makes sure the work gets done?

Does the project really matter to them?

For a business proposal, one might assume that the desire to get funding will be met with an equal desire to make money for investors. In funding a non-profit project, one would like to know that the same desire leads them to deliver tangible benefits to the people they serve. In the end, we want to know that a team is committed to the project and can be depended upon to deliver.

Collaboration is a requirement.

Collaboration is what technology enables. Some projects seemed to consider collaboration as an afterthought, and not incorporated into the development process. Collaboration is what drives the open source model.

Collaboration is often made necessary by constraints. (“I can’t afford to develop my own solution, so let me see if someone else has done so.”) Figure out early on what you can’t do yourselves and what you don’t know. Admit the limitations you’re working under but use them for effect. In a sense, this is why crowdsourcing can be valuable. When you can’t afford to buy the expertise you need, you try to find a way to leverage the community to provide it for you.

It’s also good to reference a community of practitioners who may be doing similar things. You should explain how your work will inform that community and extend what’s been done already.

Find out how you can tap into existing communities to provide additional resources or expertise.

Prepare to explain how you can broadly share what you’re doing and what you’re learning.

Leverage existing hardware/software and support free and open alternatives.

Projects that require funds to buy hardware — from laptops or netbooks to Android phones or GPS devices — are carrying an unneeded burden. Judges should question a budget that contains expensive licenses for commercial software for 20+ kids when open source programs exist that provide the same functionality. The costs for giving limited funds to hardware and specialized software can be hard to justify.

Ideally, a funder or some other source could organize a list of hardware available at steep educational discounts or even donations from a list of participating vendors. There’s really no one doing that kind of coordination. Instead, the budget says 30 kids need 30 laptops, which the kids do need to participate in the project because laptops are not already available in the schools. This might be reasonable in many cases, but then you have requests for 30 cellphones, etc.

There are ancillary issues, such as the support and maintenance of hardware, as well. It’s also not clear what happens to the equipment after the period of funding.

Avoid choosing cutting-edge technology that schools do not have. An iPhone/iPad or a gaming environment that requires more powerful PCs are examples of burdensome technology requirements that can hinder a proposal, no matter how exciting the technology opportunity. It’s difficult to fund a project that requires specialized hardware that is beyond the reach of school budgets.

Could the hardware/software be donated to your group, once you got the project funded?

Find out if other sources of funding are available to pay for hardware that can be used across multiple projects.

Produce specific results and share them.

A project should be expected to produce value that survives the end of the project. Be really specific about what you’ll produce and in what form you’ll share it. Please share openly what did not work so others don’t repeat the same failure.

You might simply write a report that details what was learned — a report not just for the funder, but for the community to learn from. You should provide data, not just a narrative.

Each project could also build or produce code that could be reused by others. If you design for reuse, others can benefit, which might include your next project. It may be only a small part of a project, but writing a module that continues to serve others in the community is a good thing.

This also applies to curriculum. Some proposals said they’d produce new curriculum but it wasn’t clear what form the curriculum would take. Is it on paper? It is online? Is it in a wiki? Is it a PDF? We should expect that new curriculum can be used by others and that can it be easily modified for new uses.

A good match for public good.

Technology is seldom the main reason that a proposal is accepted, and that is perhaps is as it should be. There’s some idea of “good match” that carries the day. That is, the proposals that win are believable in that the people or institution making the proposal can deliver and serve the selected audience, which is worth serving in terms of creating a public good.

All of this can be quite vague and rather frustrating to pin down. But it’s a form of match-making. If one looks at the team and the characteristics and needs of the people who are the beneficiaries of the proposal, and there’s a good match, then one can believe in the promised benefits. The technology — or the method of delivering the benefit — is significant, but it depends on having a good connection between the team and the people the team serves. This is another way of saying that the customer matters most.

The technology might even be peripheral to the proposal, like Hitchcock’s notion of a MacGuffin. (He defined a “MacGuffin” as an object that built the drama and moved the plot forward but was on its own insignificant.) Technology can be a catalyst for changing processes and introducing new ways of thinking, and that may be its real value.

Understand what drives usage and engagement.

Just because you build something doesn’t mean you know how to get others to use it.
This is hard, especially for education. A project that promises wide usage in theory can fail in practice if the project cannot easily recruit users.

Learn from good working examples.

Build with reference to good design practice. If you were building a website, you’d consult examples of good web design. If you build an ecommerce site, you have to be able to talk about Amazon and eBay. How do they work, what do they do, whom do they serve and how well do they accomplish what they promise to deliver?

Can we identify a group of “keystone” projects that use technology in education effectively and that can be referenced as a set of best practices?

Solicit real feedback. Repeatedly.

Getting good feedback early on is important. I’m not talking about evaluating whether the project will get funded. I wondered if the project proposals had been through rounds of review that questioned assumptions and looked into the implementation plan in detail.

(This competition had an open public review process, which consisted of the applicant’s friends and colleagues writing: “Wow. What a fantastic learning application.” This is not useful feedback.)

I wondered what might change if the judging process was done in the open. Would it have been beneficial for the teams to hear the criticism directly? I suspect that little feedback gets back to the applicant — they are simply not chosen. In a public forum, a number of comments would not be expressed by judges (such as “X university never does this thing well”) but it would be advantageous if there was more public feedback, which might raise the stakes for the competition. More teams could learn about why their proposal was not selected, and use that feedback to improve in the future.

Have a real point of view.

Some project proposals seemed plain vanilla, lacking a point of view. In other words, the project proposal described what they’d do without covering why they want to do it. I liked when a member of the team was a strong advocate for the work they were doing. Often one’s point of view relates to a much broader context for understanding the project.

As a judge, I discovered my own point of view and I looked for projects that shared this point of view. I discovered that the projects I liked had a few things in common. I preferred projects that offered a direct engagement with individual kids and that involved them in producing something of value based on their own interests. I like to see kids engaged in doing something real, not simulated. (Other judges have different views, and we each favored certain attributes, but I was happy to learn that I had begun to develop my own criteria, partially based on my experience of seeing students as makers.)

For instance, I liked a program working with 30 kids in an inner city to develop Android applications. There was a video of a teacher who was very excited about the opportunity to engage kids in how they would like to use mobile phones and what applications they’d like to see. She said in no uncertain terms that the idea of creating applications for a cellphone would allow her to reach kids who would otherwise not be interested in programming. I don’t know if they will succeed in developing useful applications, but it was more interesting to me that they try.

Knowing your own point of view will help you understand the value that you’re trying to create and connect to others who can support your work because they share similar views.

dnrta, sry too long, but how bout some good ole’ fashioned operational efficiency? like, a system that allows students/teachers the ability to communicate about things like missing class to make up work in another, for example? schools are horrible time-sinks.

Marie Bjerede

One tool I’ve seen used to good effect is the Logic Model. There are any number of variations on these, but they all provide a big visual representation of the theory of the project/intervention that is easy for stakeholders to talk about and that surfaces the assumptions and “why” of the project.

There are different structures, but in general they include:
– Inputs (or resources needed and activities/work)
– Outputs ( our outcomes of the project on the participants)
– Impact (how the landscape changes over 10 years or so because of the outcomes)

According to the W.K. Kellogg Foundation Handbook, “Most of the value in a logic model is in the process of creating, validating, and modifying the model … The clarity of thinking that occurs from building the model is critical to the overall success of the program”

Maybe a framework for talking about and sharing and modifying theories within a community?

Steve Ediger

Dale,

This article makes a great foundation for more serious studies in this area. If you put it into a wiki somewhere, I’d be happy to help develop it and circulate the link to others that might be interested.

The rubric for project development and evaluation could be the start. I also see the possibility of a repository of projects and their relative successes and failures. (I realize that I’m violating your first point here because I haven’t actually searched for edtech project repositories; they probably exist.)

Dale Dougherty

Steve,

That’s a good idea to put the article up in a wiki. Can you recommend an educational wiki with related content or should I just set up something on pbwiki?

Excellent, excellent advice, Dale. #11 strikes a chord with me as I think educators could benefit from a comprehensive database of best practices in educational technology. Where are we collecting evidence that supports our belief that technology can enhance learning? Edutopia comes to mind as a start, but I’d love to see something developed that’s more focused on technology-based materials and projects.

Dale Dougherty

Larry,

I’m happy to post in 4dmo.com. Would you recommend placing it as a link off the home page or under a category such as Web 2.0. I’m not sure how to best fit it into the site.

Dale Dougherty

Lucy,

We actually need to develop “best practices” from an educational point of view for tech developers as well as from a technology point of view for educators. What’s interesting is we have two different perspectives, but we’re looking to create a common ground for appreciating the best work — a good use of technology that utilizes effective learning strategies.

Paulina

Dale,
This article will help me to create my project.Something like effective learning strategies in on-line courses.I’m looking now on the Logic Model tool, that Marie suggested.The Steve’s idea of a rubric for project development is very important too.The third great idea was to post it in a wiki.I would like to share with all of you my project, from the beginning.Could be possible if I write it in Spanish.

Avlesh Singh

Hi!
I wanted to tip on an awesome education/research tool. It is called Webklipper (http://webklipper.com). This tool lets you highlight sections of webpages and annotate them. Once can share an original webpage overlayed with these highlights and comments.

The problem is that most of these people making these proposals haven’t actually had a chance to try using these kinds of technologies in practice.

People with some idea on how to improve education with technology are a dime a dozen. People who have actually created and used a technology and gone through many iterations of improvement AND gotten results is extremely rare.