That’s the question the Knight Foundation has been asking about perhaps the most prominent program supporting media innovation, the Knight News Challenge. Since 2007, the competition has attracted thousands of people with ideas for an innovation project, and Knight has funded more than a hundred of them. Some have grown into successful, widely used tools; others have disappeared with barely a trace.

On its face, the document reads like a lesson plan for journalism innovation, especially if you’re interested in developing a project you have to have Knight fund some day in the future. (Now would be a good time for a disclosure: Knight is a financial supporter of Nieman Lab, though not through the News Challenge.) Knight, working with Arabella Advisors, surveyed News Challenge winners from 2010 and 2011, looking at their original applications, metrics around their projects, and other materials to see what behaviors and characteristics correlated with success.

Over the time period being studied, the News Challenge featured 28 projects like FrontlineSMS, iWitness, and Zeega, among others. The document’s worth a read to get an update on how those projects are faring today — some continue on, others have spun into something new, and others have closed their doors. Knight offers eight lessons for future innovation in the report; here are a few of the highlights.

Understand your staffing needs early

Many News Challenge projects are developed by people who already have a full-time job elsewhere, which means figuring out the best ways to devote time and energy to an idea can be tricky. Having a core staff from the outset is important for putting a project on the right footing, but so is identifying what things can be done by part-timers or volunteers, the report says. You can’t just assume that your idea will find the right community of users and take off on its own:

Many projects plan at the outset to rely upon a dedicated user community to refine and promote an innovation, and upon vocal evangelists to drive wider adoption of their tools. In many cases, user communities and evangelists can become indispensable (and inexpensive) cornerstones of a project, especially when a project is dependent upon open source development. But without a core group of paid staff with the skills, the time, and the incentive to devote themselves full time to a project, development of a tool can suffer.

The pros and cons of going open source

One of the original requirements of the News Challenge was that all projects open up any code or other resources they develop. The idea being that the money invested in Knight would help not just the grantees, but potentially a wider community of developers. As the report notes, in the case of DocumentCloud, that success came through the creation of the hugely successful JavaScript library Backbone.js, now used on countless non-journalism projects across the web.

But News Challenge winners reported that the open source requirement could also slow down their efforts, especially for groups working on existing software or tools that receive funding. The report suggests that more flexibility be offered in the what projects are made open source in the future.

In some cases, the News Challenge winners themselves benefit from using and sharing open source code. In other cases, it is the wider community of developers that benefits most. It is entirely conceivable that the winner might bear the cost of developing open source code, without receiving an equivalent or offsetting benefit, which might accrue to someone else entirely.

Fundamentally, unless an innovation addresses a pressing need, journalists and news organizations will not adopt it. In fact, innovators need to anticipate resistance, and create development and marketing plans that address it. Innovators may need to diversify their user bases beyond journalists and news organizations to promote wider adoption and project sustainability.

Aim for a broader audience than you might anticipate

So convincing journalists to try something new is hard — that’s why it’s important look at what other users might have a need for what you are building. CityTracking had anticipated an audience of journalists for its public data display tools, but later focused on specifically serving developers. Game-o-matic, which lets reporters build games based off the news, shifted gears to reach users outside of newsrooms.

Pushback can come from multiple directions

Newsrooms with tight budgets and established ways of working aren’t the only source of resistance. In many cases, News Challenge projects found themselves bumping up against established institutions, and some institutions don’t like to be disrupted.

Get the user interface right early

A lesson from the world of apps and game builders, applied to journalism entrepreneurs:

User interface can play a major role in determining whether a media innovation is actually adopted by its audience — an interface that’s fun to use or saves the user’s time can make the difference between a tool that’s used and one that gathers dust. Among the innovations developed by News Challenge winners, the most effective interfaces frequently have been those that appear simple or straightforward.

Everyone needs a little support sometimes

One thing Knight has heard from News Challenge winners in the past was repeated in this report: They need more support, beyond funding. Because Knight has such a large network that touches into media, technology, government, and more, grantees said they want to be able to tap the expertise of others when it comes to developing their business plans and other strategies. In particular, the 2011 and 2010 winners said further connecting with other winners of the News Challenge would also prove helpful to learn from their experience.

Success goes beyond the impact of a single project

The best barometer of success isn’t the outcome of individual projects but the effects projects may have on their sectors or industries. Funders should focus on building the capacity of innovators as leaders in their fields or strengthening their network of supporters and collaborators for long-term impact — regardless of the sustainability of particular projects.