is the Executive Director of the Digital Public Library of America

Menu

Enhancing Historical Research With Text-Mining and Analysis Tools

February 4, 2008

I’m delighted to announce that beginning this summer the Center for History and New Media will undertake a major two-year study of the potential of text-mining tools for historical (and by extension, humanities) scholarship. The project, entitled “Scholarship in the Age of Abundance: Enhancing Historical Research With Text-Mining and Analysis Tools,” has just received generous funding from the National Endowment for the Humanities.

In the last decade the library community and other providers of digital collections have created an incredibly rich digital archive of historical and cultural materials. Yet most scholars have not yet figured out ways to take full advantage of the digitized riches suddenly available on their computers. Indeed, the abundance of digital documents has actually exacerbated the problems of some researchers who now find themselves overwhelmed by the sheer quantity of available material. Meanwhile, some of the most profound insights lurking in these digital corpora remain locked up.

For some time computer scientists have been pursuing text mining as a solution to the problem of abundance, and there have even been a few attempts at bringing text-mining tools to the humanities (such as the MONK project). Yet there is not as much research as one might hope on what non-technically savvy scholars (especially historians) might actually want and use in their research, and how we might integrate sophisticated text analysis into the workflow of these scholars.

We will first conduct a survey of historians to examine closely their use of digital resources and prospect for particularly helpful uses of digital technology. We will then explore three main areas where text mining might help in the research process: locating documents of interest in the sea of texts online; extracting and synthesizing information from these texts; and analyzing large-scale patterns across these texts. A focus group of historians will be used to assess the efficacy of different methods of text mining and analysis in real-world research situations in order to offer recommendations, and even some tools, for the most promising approaches.

In addition to other forms of dissemination, I will of course provide project updates in this space.

11 thoughts on “Enhancing Historical Research With Text-Mining and Analysis Tools”

That’s awesome. I had hoped this day would come. If you can develop software that can understand texts and find meaning in them, as opposed to just extracting and manipulating their information, I think you will really be onto something. But that may require revisiting our debate about the semantic web.

I have been planning to do the same thing that you’re already doing. I definitely would like to participate in any way that I can. I am a computer scientist at RTI International specializing in databases, data warehouses and data/text mining – particularly in bioinformatics. I am very interesting in the potential of applying text mining to the analysis of large numbers of documents of all types. Keep up the good work.

great news!
Meanwhile, until this project can be download.. can someone (Steve Ramsay?) tell me where the MONK project has gone? Why their site is empty? Can I download the MONK or the NORA project and get it run on a standard PC?
Many thanks,
Shai