Category: New Media Economics

Just over 15 years ago the first big news story I worked on also turned out to be the first big news story of the digital news era. The Clinton-Lewinsky scandal broke online, with the first and biggest stories of the 13-month saga posted online before they went into print. And it wasn’t just news stories — as Monica Lewinsky pointed out in her fascinating TED Talk last week, it was hundreds of pages of raw documents, photos, audio and video.

It wasn’t the first time the talk of presidential impeachment wafted through the streets of Washington, but it is the first scandal that is nearly impossible to re-create using the first rough draft of history. Those digital-first stories — the time and manner in which they packaged and presented online, the live online discussions, the changes big and small to the wording — are almost totally lost down a digital memory hole.

When I want to add this digital innovation to my list of “clips” for my resume, I often can’t. When I want to illustrate for my UNC journalism students a first-hand account of media innovation, they have to rely on their imaginations. And when historians and The Washington Post company look back on it not just 15 years from now but 150 years from now, they won’t have the benefit of some of the most ground-breaking features that drove public opinion.

But what’s more amazing is that if you work in a newsroom today, the legacy of your born-digital content is probably at risk just as much as mine was at the dawn of the digital news era. If you run a digital news business, one of your greatest competitive advantages is going 404 every day. And if you live in one of the communities served by a newspaper with a website, the story of your generation is being written in digital sand.

To help fix that, the National Endowment for the Humanities is sponsoring a survey of North Carolina news organizations to collect information about how they use and preserve born-digital news. It is part of the “Dodging the Memory Hole” work led by the Educopia Institute and the Reynolds Journalism Institute at the University of Missouri. They are hoping to launch other state-wide surveys in five more states later this year.

The results of this online interview will be used to by librarians, technologists and journalists to establish industry standards for preserving born-digital news and to propose future projects that help news organizations and communities preserve and protect their legacies.

The initial findings are going to be presented at a gathering of North Carolina’s journalists, librarians, scholars, technologists, publishers, vendors, and other interested stakeholders on May 11-12 in Charlotte. You can also register for this event at https://educopia.org/events/dmh

The Educopia Institute, a nonprofit that works with educational and cultural institutions to increase their impact, will keep the responses completely anonymous in all research results, so that you and your organization cannot be identified. The research results will focus only on aggregated findings.

Preserving born-digital news is just as important as finding a viable business model for the creation of it. This generation’s digital native must tell the stories of our time and making them available to future generations so they know how and why we made this environment they will inherit.

The trend in automation should free up the best writers and best reporters to add the how and why context that still needs to be done by humans. If I were a beat reporter at a newspaper I’d be working as fast I could to convince by editor to let a computer write the scut stories I have to write and free me up to do more explanatory and accountability reporting, or to craft beautifully written narratives.

One significant risk is that for the last decade we’ve seen “good enough” journalism growing in popularity. News organizations that continue to have a strategy of harvesting profits rather than investing in growth will no doubt cut reporters if machines can write commodity news at a lower cost.

If I were a young journalist looking for my first job, I’d be looking for news organizations that are sustaining a small margin and growing both expenses and revenues — the ones that are using both bots and humans.

The trend toward automation will result in an emphasis on the news value of impact. Mass customization is going to change the nouns in the leads of stories from the third person to the second — “investors” will become “you.”

The trick is how to make money off this. News organizations that continue to see themselves as manufacturers of goods will probably increase the volume of digital commodity content they publish and continue to drive down ad rates.

But smart content companies are evolving from a manufacturing industry to a service industry, and trying to create, explain and capture the value they provide to each client by getting the right information to the right people at the right time.

What we see now as data is as unsophisticated as what many of us thought of data when Google first made its mission organizing all of it. We think of data now as numbers in tables — scores, money, temperatures, but we’ll soon see data as behavior and content metadata. And we will see automated stories that incorporate the user’s data and the data of her social network as well.

That level of concierge news service, though, is going to come at a price for users. If we’ve seen the democratization of media this automation trend has the potential to create a world of media haves and have nots — the haves will pay premium subscription fees to get highly personalized news from bots. The have-nots will get generic news (maybe written by bots as well).

The one thing from which I think everyone will benefit is an increase in the quality and frequency of narrative writing, and of explanatory and accountability reporting.

To aid that transition I’m working on the idea that we can use digital public records to build a newsroom dashboard system that will alert beat reporters to possible story ideas. Automated Insights and Narrative Science are scaling commodity news stories. I want to see if we can lower the human reporters’ opportunity cost of pursuing enterprise stories that land with much bigger and much longer lasting impact.

If you want a pithy quote from a journalism prof. on the effect that robot writers are going to have on the job market for journalism students, here it is: “My C students are probably screwed. My A students are going to do better than ever.”

To publishers — trying desperately to make money by selling more inventory of advertisements that have a plummeting CPM — the only value of online ads seems is to make them annoying enough that the audience will pay to turn them off.

The problem seems to be that there’s very little online advertising that has annoyed me enough that I will pay to turn it off. I will grumble as I chase “click to close” across the screen. I’ve seen people resize their browser windows to remove banner ads from their field of view. Now ask me to name some of those ads I spent chasing across the page? I dunno. A liquor brand maybe? Auto insurance?

But this week I paid for the the first time to turn off ads. I’ve been trying out the TextPlus app on my iPad. It allows me to send and receive SMS messages to an iPad, iPhone and Android phone. The app displays ads inside the app and embeds a little self-promotional message at the bottom of each outgoing SMS.

The banner ads didn’t bother me, but it’s a little tacky for each of my recipients to see “-Sent FREE from textPlus.com” at the bottom of every message. So when I saw for $3 a year I could remove ads, I made one click to remove those little buggers. The transaction was seamlessly handled through iTunes’ in-app purchase feature. (Now, unfortunately, I failed to read the not-too-small print and I only turned off the tolerable in-app ads. The in-message ads are still there and can’t be bought out.)

So GOGII, the company that makes textPlus, got my $2.10 (Apple got the other 30 percent of my purchase) but they lost the ad inventory. So GOGII must think that’s a good trade-off, right?

TextPlus says it has 8 million active users a month, according to TechCrunch. Those 8 million people send 15.6 billion messages a year via their textPlus apps, which averages 1,950 SMS messages per year. I’m going to make the (probably incorrect, but close enough) assumption that a new ad is served with each message. So, in GOGGI’s estimation, my $2.10 is worth more than 1,950 ad impressions it expects me to generate during my ad-free year. For what it’s worth, that’s a $1.08 CPM.

But the only reason I paid my $3 is that I thought — wrongly, as it turns out — that it would remove something that I found annoying; that it would improve my user experience. But EVERY other ad-supported app and Web site I use has a user experience that is good enough for me to save my $3 to remove the ads.

On the other hand, I pay $6 a month for my DVR service — partly so I can skip TV ads.

Of course, I’ll pay for content that is superior. And if I need it for my job, or I can use it to make wiser purchasing decisions, or if it entertains me. But, otherwise, you’re going to have to annoy me into submission. That means interrupting my time. Put up a wall, don’t let me over it or around it until I pay or until you get me to agree to buy into the timeshare condo.

News organizations — newspapers — have been long accustomed to earning about 80 percent of their revenue from ads. As a comparison, Pandora gets about 87 percent of its revenue from ads. That just doesn’t seem healthy to me. Too many eggs in one basket. Are the ads not annoying enough for people to turn them off? Is the content so commonplace that annoying ads would drive the audience elsewhere? Is the service and experience of Pandora not satisfying enough that it would be fatally marred by more annoying ads?

As ad inventory continues to climb, it becomes a tough game to sell that infinite real estate. Now, of course, some real estate — The New York Times — is more exclusive than other real estate — The Huffington Post — but all properties are going to have a hard time keeping their ad revenue stable. For companies with superior content, a paywayll or service-based revenue model are going to become important replacements for a falling ad line.

I think this portends a real shift in our media consumption and a possible division in media culture and civic knowledge. There will be the news and information products: those that are annoying and free, and those that are paid and efficient.

People who have more time than money to spend on hard news will suffer through the ads to get to the free content, which may or may not be junk. People who have more money than time will pay for convenience or quality of information. As a journalist, I’d rather get my paycheck from a news organization that gets a high percentage of its revenue from subscriptions. Those organizations will be able to pay for quality reporting and editing. And the free, ad-driven sites will happily repeat that reporting to an audience that’s willing to stand on the shore and wait for the arrival of the ripple of news.

I posted an earlier draft of this proposal already, but here is the full version of the proposal that has been selected as one of the finalists by the Knight Foundation in their 2011 Knight News Challenge. Let me know your thoughts. Thanks to everyone who has helped spawn and cultivate this project. Every conversation has me more excited about what we’ll be able to do for rural newspapers in North Carolina and across the country.

Describe your project

We will build a not-for-profit clearinghouse of data from state, county and municipal governments in North Carolina and deploy them through pilot OpenBlock installations at the websites of nine rural newspapers in the state. The datasets will include public records of particular interest, such as crime reports, real estate and restaurant inspections.

We have already conducted research, funded by the McCormick Foundation, that indicates deployment of OpenBlock on the websites of small and mid-sized papers could provide significant digital revenue potential — given the interest readers have in understanding their communities. But that the main barrier to implementing OpenBlock is a lack of technical expertise at small papers as well as the high cost of ongoing data collection.

At the end of our 27-month funding period, we will have reduced the costs of acquiring, aggregating and publishing public data at community newspapers. We will also have developed one or more revenue models that demonstrate how meeting the information needs of a community can also be good business, even in small towns and rural counties of fewer than 75,000 people.

Rural news organizations are struggling to move to the digital age in part because their staffs are so small that they don’t have the capacity to identify, digitize, re-aggregate and map all the various public records available at the state and local level into databases that can be accessed intelligently by both reporters and the public.

This project tackles the lack of capacity at rural papers from two directions. It will create a centralized clearinghouse of state, county and city schemas and datafeeds that could be easily used in OpenBlock. It will also create compelling editorial content that will draw new, young readers to community information presented in a format and medium they want. Audiences for this kind of editorial product are loyal. They generate repeat visits by returning to seek updates on crime especially. And they also generate page-views and increased time-on-site as they search and sort the information.

We expect through this project to lower the costs of data acquisition and organization through a variety of methods that we will be able to assess and compare. In some cases, volunteers will pick up CDs of data from county offices. In others, journalists may scan and upload PDFs of hand-written police incident reports. In still other cases, people would key into a database the information on those PDFs. This job is so big that no single small news organization could do it. But with the support of a not-for-profit organization that provides centralized technical, editorial and advertising expertise, we could create a model for gathering valuable public records from rural America. To individual communities, these records are necessary to foster an informed civic dialog and healthy economy. But in aggregate, these records may also be able to shed light on trends in rural America that would otherwise go unreported.

This project will demonstrate one way that universities can support and advance journalistic activity – by providing a launchpad for new ventures that draws upon broad faculty expertise and student workers to lower the costs of professional, independent public affairs journalism and by absorbing some of the risk associated with new editorial product development.

Knight funding will get us off the ground and put us in a position to be a self-sustaining not-for-profit company, serving North Carolina journalists and citizens and providing a model for other states and regions to adopt.

Improving Delivery of News and Information to Geographic Communities

In small towns and rural America, the local newspaper is more than just a source of information and an engine of commerce. More importantly, it fosters and builds geographic community and sets the agenda for public policy debate. By making public records readily available and well-organized, we will support decision-making and accountability in local and state government.

This project most clearly improves the delivery of news and information to geographic communities by helping rural community newspapers make the transition to the digital age and remain relevant for younger audiences that are less informed and engaged in their own communities.

We expect several community newspapers to incorporate crowd-sourcing – a technique once known in their newsrooms as “neighbors editors” – into the process of data acquisition. Where this happens, we expect an increase in civic and community engagement. — first, by forming a network of knowledgeable volunteer citizen-journalists and by creating greater demand for truly open government records.

Unmet Needs

In many cases, data that is readily available in GeoRSS or at least CSV format from big cities is simply not available even in print from rural governments. For example, journalism students at the University of North Carolina working last semester to gather and organize public records in two rural counties for an OpenBlock application met with a number of obstacles – ranging from significant photocopying fees to inappropriate redactions and denial of access to public information.

Even when acquisition of public datasets is relatively simple – for example, public health restaurant inspections — someone must request that data from a specific county be exported in fielded data format. It is inefficient for each rural news organization to make separate requests for this data in each of North Carolina’s 100 counties. In these cases, our centralized organization would outline an initial request for the data for each county.

When Rick Thames, the editor of The Charlotte Observer, reviewed our proposal, he offered his enthusiastic endorsement. “There is no question but what this would fill a need,” he said. “Small papers can’t do this sort of work on their own. So, sadly, it just isn’t getting done. What a gift this would be for those communities. A very worthy effort that would be warmly received by the editors and publishers of every small and mid-sized paper that I know.”

What’s New?

Currently there is no tool or service that can efficiently gather, format and publish public records on rural news organizations’ sites. In part, this is a technology problem that may soon be overcome with the alpha rollout of OpenBlock later in 2011. But a much bigger piece of the problem is the data itself – neither OpenBlock nor any other technology has the ability to obtain public records as fielded digital data and create a newsworthy user interface for all the various types of records that a news organization might need.

Without a project like this there is no indication that OpenBlock will be a viable option for papers in rural communities.

What Will Change?

By the end of the project, we will have …

1. About 95 up-to-date feeds of local government data in standardized, fielded formats such as GeoRSS. These feeds will be available under a Creative Commons Attribution, Share Alike license. By providing public information in this format, we will lower the barriers to North Carolinians interested in researching trends or patterns in public policy and we’ll provide the raw material for the development of mashups or entrepreneurial applications we haven’t even thought of yet.

2. Nine community newspapers using OpenBlock to publish fresh, local government data to their audiences. These newspapers will be on the frontlines of a statewide effort to get complete and current government datasets in open, machine-readable formats. They will demonstrate multiple approaches to implementation that will be relevant to others’ during the broader roll-out.

3. Identified new revenue opportunities structured around the presentation and analysis of this data that will support their journalism.

4. Journalists and citizens interested in public policy issues will have a new tool for analyzing trends and patterns in rural issues such as environmental stewardship, public health, crime and justice, education, and economic development. Community newspapers will be able to more easily compare the experiences of their communities with the experiences in other places across the state.

5. A cost-effective model for building similar independent, not-for-profit data repositories in other states.

6. Most importantly, we will have raised public awareness of open government and we will start seeing rural counties and towns publish public data in standardized, machine-readable formats on the Web.

Why are you the right person?

This project would be tested in North Carolina and rolled out nationally. While the Raleigh-Chapel Hill area is a hub for information technology, the state has a high percentage of rural counties (roughly 70 out of 100) and a strong tradition of quality community news organizations.

The project builds on extensive and longstanding collaborations between the University of North Carolina and North Carolina Press Association.

“WOW! This is an interesting and ambitious project and I know there will be many Carolina newspapers that will want this service,” Beth Grace, the director of the N.C. Press Association, told us. “At a time when papers have lost staff and have had to postpone in-depth/investigative and trend reporting, this could bring some of that information back to papers and their readers. The North Carolina Press Association stands ready to assist — we can work with you to help assess what records most papers –and importantly, their readers — would want.”

This project will address a critical need that’s been identified through the work of UNC Knight Chair Penny Muse Abernathy with three rural papers, and in partnership with professor Ryan Thornburg, whose students have already begun collecting digital public data in these rural counties. The project was funded by the McCormick Foundation to develop sustainable business models for community news organizations.

“Our newspaper has worked with Ryan Thornburg for the past year as we try to figure out how to take advantage of OpenBlock for Whiteville.com,” said Les High, the editor and publishers of The Whiteville News-Reporter. “As is the case in most rural communities in this state, the public information we plan to display is not readily downloadable to the site. This project would provide an important community service to residents of all of North Carolina’s 100 counties – bringing the benefits of the digital highway to even the most remote areas. And just as important, OpenBlock could well be an important source of new revenue for community newspapers everywhere. This is a very important first step in making OpenBlock economically feasible for small papers to implement and use.

A database of local information – and we believe OpenBlock is the best solution at this point — is a central component of the financial strategy in the digital age. Yet, the obstacles in collecting and digitizing loom as a barrier to successful implementation.”

What tasks/benchmarks need to be accomplished to develop your project and by when will you complete them?

The project has three phases, each with its own tasks and benchmarks. We have developed a detailed timeline and budget that are available immediately upon request.

Phase I is underway with funding from the McCormick Foundation to install the OpenBlock codebase on a virtual machine, to format, ingest and publish two datasets from North Carolina local governments, and write a public report on the technical risks of the project.

The report and any code we develop will be shared with the OpenBlock community. We will publish this report by April 15. (We understand this is before funding would be available from the Knight News Challenge.)

This summer, with Knight funding, journalism students and community newspaper reporters around the state will conduct a census of public records. We will pay participants to complete forms describing the location and characteristics of state and local datasets.

At the completion of Phase I, we will publish a directory of the datasets and a report that describes the economic cost to journalism of governments not publishing data in machine readable format.

Phase I will end September 30, 2011.

Phase II: The focus in this Phase will be on reducing the costs of deploying OpenBlock at rural papers as well as the costs of acquiring, organizing and maintaining data feeds that can be easily integrated into the OpenBlock application.

By the end of this Phase, we will install eight additional OpenBlock sites, publish relevant data feeds and make them freely available under a Creative Commons non-commercial, share alike license.

We will also design, test, iterate and document sample data-collection processes for a variety of scenarios we expect to encounter during the statewide deployment of OpenBlock installations. The documentation will be critical to news organizations across the country as they plan and budget their own efforts.

Phase II will run from October 2011 through September 2012..

Phase III: During the final phase of the project we will focus on generating for community newspapers revenue models that will be used to support and encourage the continuing maintenance and development of our OpenBlock installations.

During this phase we will begin a phased, statewide rollout of OpenBlock to community newspapers and we will have a comprehensive, statewide collection of public records feeds available from our clearinghouse.

Phase III will also see the incorporation of a not-for-profit organization that will house the project after the end of the grant. It will be funded with the annual membership fees from community newspapers at which we have installed OpenBlock. This organization would also maintain the clearinghouse of public data, some of which may come from places where we don’t have media partners. Finally, it will provide editorial guidance to anyone interested in using the data to create their own data tools or to write stories about trends or patterns revealed by the aggregated data.

Phase III will run from October 2012 to August 2013.

How will you measure progress?

We will measure progress primarily by meeting our benchmarks on deadline and within budget. We will recruit partners, successfully install OpenBlock at community news websites, and collect and distribute feedback from partner newspapers.

We have developed a detailed timeline and budget that are available immediately upon request.

Ultimately, we hope to see a statewide movement to support laws and systems that make government documents and data more easily accessible to North Carolina citizens. With those public policy shifts, we believe we will see more and better public affairs journalism as well as faster and more equitable resolution of civic debates.

Do you see any risk in the development of your project?

The risks of our project fall into three categories: data acquisition, data management and publication, and revenue generation.

Data Acquisition — The goal of the project is to reduce the cost of acquiring current and complete local government data in small communities. The costs now make widespread deployment of the OpenBlock application prohibitive for small publishers.

Challenges to low-cost data acquisition are technical, political and legal. The technical problems are all surmountable – at some cost, perhaps higher than we hope. In our early going, we anticipate many data sources that will require manual entry. The risks with these data sources will be accuracy and efficiency. We hope to test various quality assurance methods across our nine initial sites.

But the real challenge we believe will be reticent government agencies and uncooperative vendors with government that make their money through government contracts for digital data storage and management.

Our experiences with student efforts to collect digital, fielded data in rural communities give us a pretty good idea of the type of challenges, if not their scope. For this project, we intend to employ reporters within each community to leverage their community-based knowledge and relationships to help overcome these challenges.

Through conversations with attorneys for the N.C. Press Association, we don’t see any legal reason that we cannot gather the data we need for the feeds to be editorially meaningful.

Data Management and Publication — This project depends significantly on the successful alpha launch of the OpenBlock installations at The Columbia Daily Tribune and The Boston Globe. We anticipate these Knight-funded launches to happen late Spring 2011.

To mitigate this risk, we have already consulted with developers to help us more clearly see the technical challenges that might stand between data collection and our goal of deploying the OpenBlock application at nine community newspapers by the end of August, 2012.

As we understand it now, the technical challenges involve scraping data, developing locally meaningful schemas for various datatypes, the development of a simple user-interface for data editing on the backend, and customizing the front-end look and feel of OpenBlock to match the websites of existing community newspapers, many of which use the commercial TownNews CMS/service.

To ensure this is adequately addressed by those with sufficient technical experience to assess and solve these problems, we will hire a qualified and cost-effective group of developers to help us.

Revenue Generation – Community newspaper publishers will participate in this project only if we can demonstrate a positive return on their investment. While Foundation support is essential to the launch of this project, sustaining it will only be worthwhile if we can help small newspapers generate revenue. On the cost side, we quickly discover the most efficient strategies for data acquisition and maintenance. On the revenue side, Penny Abernathy has been working with three small newspapers to develop sponsorship models that we believe will yield enough revenue for publishers to justify the annual cost of the service.

How will people learn about what you are doing?

There are three critical audiences for this project. First, is a national audience that we will reach through trade websites and conferences as well as the OpenBlock community that is being so well nurtured by OpenPlans and its Knight-funded efforts.

In North Carolina, we have a statewide audience of newspaper publishers, editors and engaged citizens. Our affiliations with the N.C. Press Association, the N.C. Open Government Coalition, and the School of Journalism and Mass Communication at the University of North Carolina will help us identify partner newspapers and datasets that are editorially significant.

Our most important audiences will be the local news site users and advertisers. We expect and need these citizens to become consumers of public records and advocates for digital, fielded government local government data. In many cases, we also expect that these audiences will also be our collaborators and key elements of the data collection workflow. For these audiences, the local newspaper partners will be our most important channels of communication.

Is this a one-time experiment or do you think it will continue after the grant?

The information needs of our state’s communities will be best served if this project continues beyond the term of the grant. Our application anticipates that and asks the Knight Foundation to help us create a sustainable not-for-profit organization that will be self-funding at the end of the grant.

But even if one of the risks we’ve outlined prevents us from creating a self-funding not-for-profit, the journalism community at the end of this grant will have several hard deliverables that will be used to guide further efforts:

A description of state and local datasets in one of the nation’s most populated states. (August 15, 2011)

A Paper that describes the economic cost to journalism of governments not publishing data in machine-readable format, compared to the costs of the governments – and taxpayers – to do so. (September 30, 2011)

A clearinghouse of state and local government datasets, in open, machine-readable format. (September 30, 2012) A handbook of data collection processes suitable for six different public records request scenarios. (September 30, 2012)

Nine installations of the OpenBlock application at community newspapers. (July 2011 to August 2012)

Scraperscripts, schemas and other contributions to the OpenBlock Project. (April 2011 to August 2013)

The value of journalism programs today is not that we place students in reporting and editing jobs, but that we teach them how to think journalistically (as I’ve written about before here and here.) One way we can expand journalistic thinking to the entire campus community as well to amateur journalists is to help our students form a peer-editing corps. Journalism students internalize their classroom learning as they explain it to others. The quality of amateur journalism increases and the curiosity and precision required by journalistic thinking becomes part of our campus culture.

Students studying news writing, reporting and editing would hold periodic peer-editing sessions with a small group. Perhaps three journalism facilitator-students to 10 participants. Journalism students would critique the writing and reporting of the participants. Is the writing precise and concise? Is the spelling and grammar accurate? What questions are left unanswered?

An “each-one, teach-one” approach such as this would cost next to nothing. A few thousand dollars a year to support the role of a faculty mentor and various community-building activities.

Assessing the effect of a program like this would be easy as well. Survey all participants. Analyze content of participants before and after. Compare participant’s blog posts with non-participants blog posts.

It incorporates research, teaching and community service.

Some questions:

* Would there be interest among journalism students?

* Would there be interest among non-journalism students?

* Would there be interest among amateur, volunteer journalists in the community?

* How would the program reach volunteer journalists in communities far from campus?

* If you’re an employer, how much do you think an applicants participation in a program like this — as a mentor — would influence your hiring decision?

* Are there similar programs out there that already exist?

* Could you incorporate this as service-learning component of a basic news writing/editing/reporting class?

At the top of my To Do List this week is the completion of one of the proposals I’ve submitted to the Knight News Challenge this year. I’m posting it here in the hope that you’ll have some feedback on whether/how a service like this would be technically feasible. editorially useful and financially viable. I’m especially interested in hearing from editors of small papers, public records experts, civic/community organizers and anyone who’s worked on the OpenBlock code.

Under what conditions would you volunteer to help a project like this in your community? News organization — how much would you pay for a service like this? What characteristics would it need to have to make it worth your money? What else do you see here that needs further clarification?

(And a big hat-tip here to Penny Abernathy, the Knight Chair in Digital Media Economics here at the UNC-Chapel Hill School of Journalism and Mass Communication. She got this project kicked off with a grant from the McCormick Foundation and who is my co-pilot on this application.)

Here’s our draft pitch:

Crowdsourcing Data to Bring OpenBlock to Rural America

This project would create a co-op to develop and deploy public records databases at news organizations, especially those serving communities of fewer than 75,000 people, preparing those records for presentation and integration in an OpenBlock format.

These rural news organizations are struggling to move to the digital age in part because their staffs are so small they don’t have the capacity to identify, digitize, re-aggregate and map all the various public records available at the state and local level into databases that can be accessed intelligently by both reporters and the reading public.

The project would tackle the lack of capacity at rural papers from two directions. It would create a centralized repository of state, county and city schemas and datafeeds that could be easily used in OpenBlock. This a job well-suited for a small group of experts. In addition, the project will create a statewide corps of amateur data-checkers and records requesters. Data quality assurance and data gathering are jobs well-suited for a crowd of many people, each working on a small piece of the puzzle.

These volunteer citizen-journalists would actually be member-owners of a co-op business. Each task they perform would earn them additional shares in the company’s annual profits. We would generate revenue by charging rural newspapers a fee. The more records and the better their accuracy, the more news organizations would sign on for the service.

In some cases, volunteers would pick up CDs of data from county offices. In others volunteers would scan and upload PDFs of hand-written police incident reports. In still other cases, people would key into a database the information on those PDFs. This job is so big that no single small news organization could do it. But with a corps of member-owners working together, we could create a model for gathering valuable public records from rural America. To individual communities, these records are necessary to foster an informed civic dialog and healthy economy. But in aggregate, these records may also be able to shed light on trends in rural America that would otherwise go unreported.

Improving Delivery of News and Information to Geographic Communities

In small towns and rural America, the local newspaper is more than just a source of information and an engine of commerce. More importantly, it fosters and builds geographic community and sets the agenda for public policy debate. This project will foster civic and community engagement — first, by forming a network of knowledgeable volunteer citizen-journalists, and also, by making public records readily available and organized to support decision-making and accountability at all levels of government.

Unmet Needs

In many cases, data that is readily available in GeoRSS or at least CSV format from big cities (such as this example from San Francisco) is simply not available even in print from rural governments. For example, journalism students at the University of North Carolina working last semester to gather and organize public records in two rural counties for an OpenBlock application met with a number of obstacles (which they describe in their blogs) – ranging from significant photocopying fees to inappropriate redactions and denial of access to public information.

Even when acquisition of public datasets is relatively simple – for example, public health restaurant inspections — someone must request that data from a specific county be exported in fielded data format. It is inefficient for each rural news organization to make separate requests for this data in each of North Carolina’s 100 counties. In these cases, our public records coop would outline an initial request for the data for each county.

What’s New?

Currently there is no tool or service that can efficiently gather, format and publish public records on rural news organizations’ sites. In part, this is a technology problem that may soon be overcome with the alpha rollout of OpenBlock later in 2011. But a much bigger piece of the problem is the data itself – neither OpenBlock nor any other technology has the ability to obtain public records as fielded digital data and create a newsworthy user interface for all the various types of records a news organization might need.

Without a project like this there is no indication that OpenBlock will be a viable option for papers in rural communities.

What Will Change?

By the end of the project, we will have

• at least one member-owner in each county in North Carolina

• at least 12 news organizations subscribing to the service

• at least one type of schema for which we’ve collected data from each county

Most importantly, we will have raised public awareness of open government and we will start seeing rural counties and towns publish public data in standardized, machine-readable formats on the Web.

What tasks/benchmarks need to be accomplished to develop your project and by when will you complete them?

How will you measure progress?

Do you see any risk in the development of your project?

How will people learn about what you are doing?

Is this a one-time experiment or do you think it will continue after the grant?

At least once a week I get an e-mail with an amazing offer for students – “write for my fledgling entertainment/product-review/sports/music/opinion Web site and be rewarded with fame, a clip and a variety of other compensation that don’t pay the rent.”

If you’ve sent me one of those emails recently, there’s a pretty good chance you’re still waiting for me to take advantage of the opportunity.

Maybe I’m missing out on the chance to make UNC the “hub of journalistic” activity in a changing news environment. After all, this recommendation from the Knight Commission on the Information Needs of the Community is really an economic one – its goal is to reduce the cost of providing news and information. And one of the ways that news organizations have always reduced the cost of production is to rely on an apprenticeship system of unpaid college students. The benefit for the student has always been that “clip” – currency that can be later traded for a paying gig.

But with those paying gigs becoming fewer and farther between, this relationship between the classroom and newsroom is going to change. The only students left taking unpaid writing opportunities are those who do it for their own psychic gratification or who don’t get the changing economics of journalism.

If universities are going to be “a hub for journalistic activity” in a world where Demand Media and Associated Content have turned low-quality “news” writing into hourly shift work, we’re going to have to be part of this economic shift in a way that benefits our students, our faculty and the communities that those of us at public universities are paid to serve.

Leadership Will Come From Research

Universities can lead change not my managing it, but by informing it – by providing the road map of answers to complex questions.

Innovation is the buzzword that we all toss around when we talk about the risks that many media companies are unwilling to take. The theory goes something like this: universities can innovate because they don’t have to show a return on their investment. They don’t have to spend resources on product maintenance.

But innovation without a problem and a hypothesis is aimless. Universities don’t need to create new stories or better stories for a market in which there’s insufficient demand. They need to formulate precise questions with answers that will impact the free exchange of ideas and support communities’ information needs.

It’s not that universities aren’t conducting research in the fields of psychology, economics, management, political science, sociology, biology that are relevant to the information needs of communities. But it seems as if there’s a failure to connect that research to environments where it can be applied.

The failures come both from the newsroom and the classroom side. Newsroom leaders, as a broad generalization, are too myopic in the questions they think are significant to the futures. Driven by incredibly demanding and increasingly frequent editorial and financial deadlines, the requests they make of universities are narrow in scope and short in term. On the other hand, academic research tends to further theory rather than change newsroom behavior. The standards of academic research FAR exceed the standards of industry research. The statistical rigor, the theoretical grounding, the editorial process of academic journals and conferences favor slow, careful dialog. We need to find a way to get “half-baked” research into the hands of industry where it’s more important to take an educated guess today than to make air-tight reviews of missed opportunities.

If I had the time, I’d love to go through the last five years of published academic research on all fields of mediated communication and create a “So What?” guide that would identify places where the direction of research can lead newsroom decisions about story choice, story presentation, audience development and information efficacy. Maybe someone’s already done this, but I’m going to bet that fewer than 5 percent of all working journalists in the U.S. today could name a single academic journal related to their field. If I’m even close to being right, then this is a foundational disconnect for which both the newsroom and classroom bear responsibility and for which the price is a an industry and a public discourse that is less healthy than it could be.

And maybe industry – and government — is already paying for academics’ time. But my guess is that the information industry invests far less in research and new product development than they spend on research in the health sciences. If universities are going to be the hub of journalistic activity then they are going to need to make research a spending priority and they are going to have to find businesses, governments and foundations willing to provide long-term financial commitments to it.

Universities aren’t going to lead the journalism industry by providing undergrads with new skills. They will only lead if they are providing a new roadmap founded in research. This is primarily the job of faculty in journalism, but also faculty in a broad range of fields with which academic journalists need to continue to seek collaboration. We can and should expose our undergrads – and certainly our grad students – to research. If we do that, then we’ll be churning out leaders who can get into newsrooms with an appreciation of research and understand how to manage its application.

Fewer Journalists, Better Journalists, Higher Pay

My UNC colleague Phil Meyer was right when he said back in 1991 we are “raising the ante on what it takes to be a journalist.”

Part of my emphasis during my four years on the faculty at UNC has been to find a way to “infuse online journalism” into courses that already exist in the curriculum. Lord knows I’ve tried and continue to do so. <plug>The most tangible evidence of my efforts is my new textbook Producing Online News.</plug>. But my problem has been this: the Accrediting Council on Education in Journalism and Mass Communication requires that at least 80 hours (out of a typical 120-hour four-year undergraduate career) be taken “outside of the unit.” Infusing new skills and concepts into journalism education is a zero-sum game. I can’t add new lessons without removing something. Or I have to find a VERY creative way of teaching the old and the new both at once.

I actually strongly support the goal of the ACEJMC requirement. Journalism students MUST have a broad understanding of the world. A broad liberal arts education ensures that a journalism degree doesn’t just mean that students have learned the trade of laying out pages in CSS or operating a camera or applying AP style. I tell my intro news writing students that a good definition of news is when the world doesn’t behave as you’d expect. And that means you have to be familiar with the natural, social and economic rules that determine how the world usually behaves. You don’t learn those rules in the J-school.

So rather than narrow the breadth of a journalism degree or reduce the importance of spelling and grammar, I’d like to see the ACEJMC introduce a new professional journalism degree that is neither an undergraduate nor a graduate degree program. The model for this would be the Pharm.D program that the American Association of Colleges of Pharmacy created in 1990 for pharmacy students.

(For those of you heard me mention this idea at the ONA conference in October, I hope this further explanation helps with your well-founded suspicions that I might have finally gone off the deep end.)

Here’s how a “doctor of journalism”, or Journ.D., program would work. Students would take the standard 80 hours of “General College” requirements, half of which would need to be in communication-intensive courses (selections in departments such as communications, psychology, politics, literature, sociology, computer science, and – yes – even journalism). Introductory course from the journalism school – such as news writing, media law & ethics, design, photography, media history, and media criticism — might even be required

This requirement would be similar to requirements at the Pharm.D. program at UNC — 40 hours of general education and 45 hours of math and science course at the undergraduate level.

After completion of the general education requirements, students would take a test and be selectively admitted to the Journ.D. program. This program, again modeled on the Pharm.D. program, would require another 80-100 hours of classroom instruction in communication skills and subject-specific reporting techniques. It would also require nine to 12 months of supervised field work. From freshman year to graduation, the degree would take six years to complete.

The supervised field work supports the Knight Commission goal of creating a hub of journalistic activity and it has a model in the Pharm.D. program. Pharmacy students at UNC are sent to work with one of 500 “preceptors” – or professional mentors – at one of 275 sites around the state. Some students also choose – at greater expense to themselves – of working at national or international sites. While many of the students want, understandably, to work near Chapel Hill, the field work is done all over the state including urban and rural environments that desperately need health care professionals beyond those that the private market alone can support. Can these communities need any less the civic capital and personal decision making tools that high-quality journalism supplies? Are a community’s information needs any less important to just public policy and rural economic development? I know from working with small and medium sized newspapers, non-profits, public radio stations, and hyperlocal bloggers that there ain’t a one of them that couldn’t use more journalistic firepower. They do, after all, come seeking free undergraduate labor…

Precpts, for their part, may or may not get some salary from universities. But they certainly all get free labor and access to the university’s faculty expertise and research. In exchange, they develop a curriculum with learning goals and give the students a pass/fail grade. And that to me seems like a fair trade.

In some cases, universities provide pharmacy students with subsidized housing at their fieldwork sites. But in other cases, the students foot the bill.

And discussions about footing the bill is perhaps where the Pharm.D. model begins to show some flaws. From my cursory research as well as from anecdotes provided by friends in both fields, starting pharmacists in any market and at any level of experience make about twice what journalists make. Pharm.D. students are willing to invest in six years of tuition payments because their return is much better.

I think Phil Meyer was right – being a great journalist today takes a lot more than being a great journalist 15 years ago. The need for advanced training is clear. But the justification for students to invest in it is pretty weak. Until the demand for high quality journalism increases and/or the supply of high quality journalists decreases, we have a pretty weak argument for requiring students to spend more time and money breaking into the field.

Universities, through research, should play a role in increasing demand. If advertising, social pressure and public policy changes can get people to exercise, quit smoking, seek counseling, stop bullying, recycle, and eat local then those same tools can certainly be used to increase demand for public policy news and information – maybe not among everyone equally, but some.

Universities can also help throttle back the supply of journalists who are anything less than excellent. The market is doing a pretty good job on its own of reducing the ranks of experienced, expensive journalists. But we owe it to our young students that they are not just cheap production inputs that the industry needs today. Every journalist student must be equipped for long-term success and inoculated as best as we can against the next generation of young, cheap, inexperienced labor.

Many of our journalism students are incredibly idealistic and would work for not much more than food. But how are we rewarding and supporting their engagement in public life by chucking as many of them as possible as quickly as possible into a market that doesn’t value them?

Looking at the success of trade publications, Congressional Quarterly and Bloomberg lead me to believe that high quality reporting that has a direct impact on its audience will always have a market.

We need to raise the ante on what it takes to be a journalism student.

Journalism Schools and We the Media

For at least 100 years, one of the biggest criticisms of journalism programs at universities is that they are anti-democratic. Who are you, pointed-headed professor, to tell us who is and who is not qualified to participate in public debate, to watchdog the government and to tell our own stories of the world the way we experience it?

And with the decreased barriers to publishing journalism, this criticism only gains traction. The distance between a non-partisan professional journalist, a professional advocacy journalist and an amateur journalist – once the audience – is smaller than ever.

More and more people are banging down the door of the MSM and the institutions that develop it. While raising the ante on what it takes to become a professional journalist, J-schools can also throw open their doors and take advantage of amateur interest to develop more and better journalistic activity and capacity in the communities we serve.

Journalism schools – perhaps staffed by some of those new Journ.D. students in their fifth or sixth years – should be running programs that teach technical – and that includes writing – skills as well as programs that advance the journalistic values of verification, precision, brevity, relevancy.

Amateur journalists, engaged in narrow and personally satisfying swaths of public life, are an important tool in filling the information needs of a community. These amateurs will report small datapoints that aren’t economically rational for a professional journalist to ferret out. Those datapoints, in the hands of professionals, can help illustrate trends and structures that might otherwise remain obscured or misunderstood.

Amateur journalists who appreciate journalistic values will also have a greater appreciation for professional journalism.

Journalism schools should be places that train and – hang with me on this one – accredit amateur journalists. I would never – NEVER – support any legal privilege that would make the First Amendment work better for one person than another. Accreditation of amateur – or professional — journalists is merely a way that can help the marketplace of ideas function more efficiently. The more accrediting bodies the better (up to, I suppose, a point of diminishing return). The market of individual choices can determine what the different accreditations actually mean. But at least give that blogger who completed nine continuing education credits in reporting, community management and communication law put a little JPG on her site. Degrees have value, and they differ from accreditation badges only in scope and scale.

Why Have Journalism at Universities Anyway?

I don’t know whether the Knight Commission intended this, but it seems to emphasize the role of community colleges as hubs of journalistic activity. Maybe I have my own insecurities about this, but I wonder if that’s not part of a broader trend in academia that historically, and I suspect increasingly, doesn’t value journalism schools, classes, departments as much as it values history, biology and mathematics. I think that many professors at research universities might see journalism as a trade more apropos for a community college.

Perhaps that’s because we in J-schools talk – a lot – about job placement of our undergrads. The students and faculty measure our own success in these terms probably as much as any other. But journalism belongs at universities not because we teach students how to be journalists, but because we teach students how to think journalistically.

Journalistic thinking approaches the world in a certain way. As I mentioned earlier, it values verification, curiosity, precision and brevity. And those values – perhaps sparing brevity – are the values of the academy. But journalistic thinking also values immediacy and proximity. A shorter way of saying it might be that it values impact. Curiosity leads to verification. That leads to answer that help people make personal and public decisions here and now.

Journalists, as the Knight Commission said, connect the university to the world. Journalists democratize intellectualism. Universities and higher education aren’t just a tool for economic development. Curiosity and verification and precise communication are tools for civic engagement – for social change as well as social stability.

For all our emphasis on changing communication technologies and changing economics of media, universities are a hub of journalistic activity when they yield curiosity, rigorous verification of fact and wisdom that has immediate impact.

Journalism is essentially applied or clinical social science. Universities should just be hubs of journalistic activity. Journalism programs should be hubs of universities.

Correction, 2/15/11: An earlier version of this article misnamed the professional mentors in a pharmacy program. They are called preceptors.

I’d be remiss if I didn’t link here to the op-ed I wrote with Fiona Morgan that appeared in today’s News & Observer. There’s been lots of good conversation on it already on Facebook and Twitter, so I’ll monitor and re-cap it here later.

I have a tumultuous relationship with breaking news e-mails. One day we have a strong relationship that I value. And the next thing I know they get all high-maintenance on me. Sheesh.

So today I unsubscribed from breaking news e-mail alerts from CNN and NPR. I kept the alert from the New York Times for two reasons:

because it does a better job than CNN of bothering me when I want to be bothered and leaving me alone when I want to be left alone. The Times’ news judgment is closer to my own than CNN’s news judgment.

because it is meatier than either the CNN or NPR alerts. All three tell me what happened. But only the NYTimes also tells me “So What?” without a click to its site. That’s especially great when I’m reading on my iPod touch and away from WiFi.

I perceived no difference in speed among the three providers. OK, so maybe one will beat the others sometimes by a few minutes. And unless it’s about an asteroid falling on my head, I just don’t care.

After thinning the herd on the national news, I planned to dump my alerts from either the News & Observer or WRAL. But when I went to do it, I just couldn’t choose. Looking over the past six months of alerts, their news judgment seems to be radically different. It’s almost as if one news organization will not send an alert if the other organization already has. So in order to get a complete range of local news alerts, I need both. But the downside to that arrangement is that probably 50 percent of the local alerts from either provider do I consider important enough to merit an interruption in my inbox.

So now what strikes me is how little time I spend talking with students about “good” news judgment and writing style for e-mail alerts. And how difficult it is to teach a technique that seems to have no consistent application among professionals. This is the perfect example where we in the classroom need to document the editorial processes around writing and distributing breaking news alerts in various newsrooms. In each newsroom, what do the journalists say are the goals of the alerts? Is there internal or external agreement on those goals? And then we in the classroom need to develop quantitative research that can help the professionals know which news judgment and writing styles best meet those goals. And then we in the classroom need to develop experimental editorial products that do a better job meeting the goals — maybe change the way news judgment and style could be tailored to the needs of individual users based on their demographics, location or behavior.

In the end, the common email alert seems to be a great example of a place where academics and industry could work together to build a better product and foster a more information society.