May 27, 2010

Evil Academic Overlords for Peer-Review Reform

As I've started copyediting the last batch of accepted manuscripts for Education Policy Analysis Archives (EPAA) from my editorial tenure, I've been thinking of John Willinsky's and Kathleen Fitzpatrick's comments about academic publishing, open access, the peer review process, and academic credentialing in general. In his incrementalist "let's push any move towards more open access" view, Willinsky pointed to Gene Glass's founding of EPAA as an example of one route to access, what Willinsky called the "zero-budget" journal. And Fitzpatrick's discussion of peer review (in Chapter 1 of the draft for Planned Obsolescence) pointed out the dilemmas of trying to generate a sustainable model of review that's new. As I'm seeing the end of my duties coming up (you really thought an editor's duties ended strictly at the end of the editorial tenure?), it's given me a chance to think about the trajectory away from subscription-based print journals. I don't know where academic publishing is headed, precisely, but I know what has happened in the recent past.

EPAA is a refereed journal, and I tried to run the English-language review process as close as I could to existing models, with double-blind reviews for the most part. But EPAA was and remains published completely open-access, free to anyone who can download the articles. So it moved one giant step away from the model of academic journals that dominated several decades after World War II, within a prepublication peer-review model. When Gene began the journal in the early 1990s, it was distributed through an e-mail list. This was only one of Gene's projects to broaden the discussion of education research through email lists, and he set up a number of lists for the various divisions of the American Educational Research Association.

He also set up a generic list on education policy, which is how we met in the mid-1990s. In a postdoctoral position at Vanderbilt, I started exploring lists and this new thing called the Mosaic browser. I subscribed to John Lloyd's spedtalk list on special education. Then I found edpolyan, which Gene had created, and I became deeply enmeshed in a vigorous 1995 debate about the Tennessee Value Added Assessment System. Eventually I started submitting articles, joined the editorial board, and was encouraged to apply for the editorial position in 2004.

In the past almost-six years, I have learned a number of things most social-science and humanities journal editors learn: how institutional support gives you some time, but never enough; how odd it is that submitted pieces can both fit the mission of the journal and leave you scratching your head on who can competently review them; how hard it is to get ad hoc reviewers to respond to requests; how review logistics are like cat-herding, only without the organization; how uneven your colleagues' research and writing skills are; how uneven your own are in comparison with some fabulous new scholars; how you never really knew how much you were avoiding learning the intricacies of a particular journal/citation style, and how much more successful some of your journal authors had been at avoiding that; how wonderful many new scholars are, and what a joy it is to give them a venue side-by-side with well-known scholars; what a great feeling it is to organize reviews so you can give coherent advice for revision; how you can be both absolutely on-target and completely off-base in predicting what articles get read, commented on, and cited; and how much you wish you could clone yourself so you could devote enough time to the journal, devote enough time to teaching, devote enough time to your own scholarship, and still have a life.

Running an open-access journal on something close to a zero-dollar budget (the college gave me a little break on teaching, and I had a wonderful graduate assistant for one year to help out), I learned quite a bit more: take the last clause in the sentence above and multiply it several times. A zero-budget operation is not an easily sustainable model to accomplish all the tasks required for a refereed journal. It requires a certain supply of surplus time, and there are no guarantees that an editor (or editorial team) will have the surplus time on a continuing basis for the central tasks, or that a reviewing pool will have the surplus time for refereeing.

Fitzpatrick addresses the reviewing part of the question, or at least the question of what would need to happen with a shift to post-publication review. She is on-target when she points out that the critical element is the evaluation of reviewing. In a standard pre-publication referee process, the editor (or editorial team) filters the referee reports, and any replacement would have to satisfy the discursive element of academic (meta-?)evaluation that Lamont described.

I understand Fitzpatrick's leaning towards an algorithm, carefully constructed, again because I worry about the time required for thoughtful moderation. My experience with the mass-reviewing process at one of my scholarly societies is not positive: I regularly receive reviewer comments for American Educational Research Association meeting proposals that are widely divergent and often enough show that the reviewer either did not read my proposal or had no clue what the standards of a discipline were. Because of the algorithm AERA uses to apportion session slots to divisions, there is a perverse incentive for divisions to encourage oversubmissions (and I've seen that operate in at least one division). That leaves program committee members the distasteful task of looking at an inflated number of submissions with divergent and sometimes irrational ratings by reviewers within a narrow window before recommendations on acceptances are forwarded from the division volunteers to the central processors of submissions. The result is that I frequently see at least reasonable proposals (both mine and others) that are not accepted, while the program has hundreds of sessions each year that are remarkably frugal in their use of scholarship. The frequent ridicule of AERA has its origins in a self-defeating program-development structure.

Maybe a more anarchic approach would work: scholars who have surplus time could become ad-hoc reviewers of working papers that appear online. I occasionally write brief blog entries on papers that are likely to gain attention from newspaper reporters, and I could as easily write entries on working papers that appear online in other areas of interest. The advantage: no one has to organize this, it would be transparent, and readers could judge the work in the context of what I write in other entries (as well as my published scholarship). It would also feed into Google's pagerank algorithm by linking to the working paper. The disadvantage: it's anarchic, so idiosyncratic public reviewing of working papers will not satisfy the scholarly credentialing process Fitzpatrick discusses. And though my blog has an ISSN, it would probably not feed into Google Scholar's algorithm. On the other hand, if more scholars are likely to read and cite someone else's work because I write about it on my blog, maybe that's not a bad thing. On the third hand, I don't really want to be a kingmaker in my subfield. On the fourth hand, maybe the fears of Sherman Dorn as Sole Public Reviewer for a certain area will push others to become more active either on their own or in creating the type of post-publication reviewing/endorsement organization that Fitzpatrick advocates.

I suspect I'm not nearly as fearsome as necessary to spur people to create such a system, but one can always dream of being an Evil Academic Overlord. Organize post-publication review or I shall destroy you!

An immodest and hopefully obvious proposal for electronic citations

I had a thought today after reading of Barnes & Noble's new iPad app, which allows customers to loan/borrow purchased books. I haven't heard whether the annotations go along with the lending, but it strikes me that academics needing to cite locations in ebooks and those interested in annotation technology both need a way to refer to locations within electronic documents.

The problem for academics looking for citation conventions is that we're all used to page numbers, which give us a way to identify a location manually by flipping through pages (or by hunting for a letter or other archival document within a file folder). Do we really need that sort of human-navigated location specificity? If we can search for text inside a document, we certainly don't. But the reference format is needed, and I think there would be an easy way to create another convention that would serve both academic purposes and ereader technology:

Given a particular edition (i.e., uncorrupted file in a recognized format with a file size and checksum), this would give a precise location. With a different edition, the approximate location within a file and the first part of the quoted passage should be sufficient for finding the passage quickly. Let's call the three numbers a brief spot location reference and the numbers plus the quotation the spot location reference. What if you're referring to a passage?

371324-375241/3/1346372044/139823463

I know I'll be torn limb-from-limb by my fellow historians, until I point out the following:

That's the range reference, the first and last ten characters of the (theoretical) passage, annotation text, annotation author, and timestamp of annotation. And there, ladies and gentlemen, is a format for annotating electronic materials. It does not require changing the EPUB format, just tracking a file of annotations and ereader software that can put the annotation in the right place (the start and end of the passage for disambiguation). They can be shared, accumulated, analyzed, etc.

There may be important reasons why this wouldn't work, but I can't think of them at the moment.

May 25, 2010

"...and thereby to secure a more arbitrary and unlimited authority"

Yesterday afternoon (at least afternoon in California, where the radio station operates), Sara Goldrick-Rab and Richard Vedder debated who should attend college on KPCC's Patt Morrison Show. I am disappointed but not too surprised that Vedder skipped over who he generally thinks are the types of people who don't benefit from college: other people's children. (Amy Slaton made a similar point in this morning's IHE column: "These two assertions [of the not-everyone-should-aspire-to-college crowd], the first based on very selective logic and the second baldly elitist, become particularly nasty in tandem, making the college aspirations of minority or poorer Americans seem positively uppity.") Let me step away for a day from the question of who should attend college today and see how that logic would have been applied in the past--discouraging formal schooling for those who would not necessarily finish a certain level and for whom there wasn't an economic payoff.

To put it bluntly, that logic would have prevented the coeducation of primary schooling in the nineteenth-century North. As David Tyack and Elisabeth Hansot explain in Learning Together, the first half of the nineteenth century witnessed a quiet revolution in formal education, from schooling being the domain of boys and men to coeducation in the first few years of schooling (which was generally what was available for most children in the North). There had been some colonial examples historians can identify of coeducation and women teachers outside dame schools, but they're the clear minority of experiences. When Benjamin Rush helped John Poor obtain a state charter for the Ladies' Academy of Philadelphia in the early 1790s, he was using his influence to break down existing barriers. Five years before, he had spoken at the new school, and the written report of his remarks starts with a justification for the school that (at least to me) looks to men as the audience, and only at the end does he start to speak to the students in the audience:

To you, therefore, YOUNG LADIES, an important problem is committed for solution; and that is, whether our present plan of education be a wise one and whether it be calculated to prepare you for the duties of social and domestic life.... I have sometimes been led to ascribe the invention of ridiculous and expensive fashions in female dress entirely to the gentlemen in order to divert the ladies from improving their mindsa nd thereby to secure a more arbitrary and unlimited authority over them. It will be in your power, LADIES, to correct the mistakes and practice of our sex upon these subjects by demonstrating that the female temper can only be governed by reason and that the cultivation of reason in women is alike friendly to the order of nature and to private as well as public happiness. (pp. 91-92)

To us more than 220 years later, this quaint and charming language obviously lacks the fire of Tom Paine and the righteousness of Mary Wollstonecraft, but for all its gentility it is an affirmation of common humanity and educability that Rush and his audience knew could not be taken for granted, even in Quaker-influenced Philadelphia. Tyack and Hansot struggle somewhat with the question of how coeducation could happen without significant public debate, and I struggle with it as well: how much to ascribe to the coeducational experience of dame schools, to early-national ideologies of Republican motherhood, to a practical "I want the girls out of my hair, too" attitude of rural Americans (who often sent children as young as two and three to tag along with older siblings), to the Second Great Awakening, or to the fact that rural apprenticeship was a system of sharing childrearing that included girls as well as boys (if the girls were often distributed to neighbors' houses to help with domestic responsibilities).

Whatever the causes, there are two undeniable facts about the coeducation of primary education in northern states: the expense was not easily justified by the legal or economic role of women at the time, and it had enormous benefits for the entire society for generations to come. I am sure Vedder and others would contest the first claim, but there are plenty of agrarian societies where the majority of work is or has been done by women who have little or no formal schooling. Why do you need to read and write if you're in the fields all day? Just go to a taro-harvester certificate program for a few weeks and get a job! Oh, wait: no community college currently offers a taro-harvester certificate.

More seriously, one could imagine a different history, closer to the history of the South, where coeducation happened much later, incompletely, or not at all. Primary education was an expense for communities, and coeducation was an added expense either for the community or for the parents who paid extra tuition (or private payments to schoolmasters on the side). We know that formal education was a considerable expense in part because even in Massachusetts, communities resisted the creation of high schools until late in the 19th century. The 1860 town vote of Beverly, Massachusetts, to abolish the high school was notable because it was a clear violation of state law (Beverly was one of the towns sued by the legislature earlier because they didn't have a high school) as well as because its public recording of individual votes has bee the subject of two books. Only a relative handful of students could continue to high school, and the majority of voters at that town meeting clearly thought the benefits of high school did not justify the expense. Yet by 1860, most towns in the north had coeducational primary schools, and thousands of parents had been willing to pay extra money (and had been willing to pay it for decades) to get their daughters some education, though the daughters would never be able to repay them in any concrete sense.

Yet despite the lack of immediate calculable returns, the coeducation of primary schooling in the North was one of the smartest social policies for the long term. The education of girls doubled the pool of potential teachers one generation later. Combined with lower fertility over the 19th century, the increased pool of potential teachers dramatically shifted the ratio of children to potential teachers in favor of children and education. Apart from arguments I could make about lower fertility's being a consequence of coeducation, the combination effectively provided a bootstrap for American mass education, making it easier for states to expand formal schooling generation by generation. Some parts of that bootstrap were not what we'd choose today, since it partly depended on restricted employment opportunities for educated women generally and educated men who were not white. But it would not have existed without primary education for girls and without the willingness of parents and communities to spend money that they could have easily not spent.

Part of the case against expanded educational opportunities is a show me what it'll do today argument. That's a narrow reading of the potential of people who don't currently attend college, a narrow reading of the purpose of education, and a narrow reading of the consequences of education. Yes, I think a lot more children from poor families can succeed in college than do currently. Yes, I want the people who pick up my garbage to read Shakespeare and pick out the lying statistician on a witness stand. And, yes, I am confident that there will be positive consequences for expanding college opportunities far into the future, consequences we cannot imagine today and that will dwarf the real costs of expanding those opportunities in the institutions where they will exist.

May 23, 2010

A hexadecimalful for hacking the academy

I do not regret not applying for THATCamp Prime (The Humanities and Technology unconference) this year, as it fell on the weekend of my anniversary, but I do miss the conversation as I woke up this morning reading the tweets (#thatcamp if you're curious), and I hope those participating in the game jam write up their notes for more public consumption. One of the side projects is Tom Scheinfeldt and Dan Cohen's call for contributions to Hacking the Academy, which they hope to collect within a week. A book in a week? I'd consider that a wee bit ambitious if I didn't know them. And I'm glad I'm not teaching this summer, so I have the time to write a short essay.

I am an incrementalist radical, certain that change can happen, good change, without enormous discontinuity. So my vision of hacking the academy is less disruptive than what others imagine. In many ways, the academy has been in the process of being hacked for decades. My own experience as a student and academic illustrates that history. I was in the generation of college students who often enough began high school with typewriters and ended college or graduate school with computers (mine: a Leading Edge XT bought when I entered grad school in 1987). In college, I took a Greek literature in translation course from someone who was a founder half a decade later of the Bryn Mawr Classical Review, one of the first online humanities journals. In graduate school I was among the third generation of social-science historians to learn SPSS or SAS; I hope I was among the last to do part of my dissertation research on a mainframe VAX. At the end of grad school, I searched for job openings using a Gopher connection to the Chronicle's job database. I created my first webpage in 1996, and I've been blogging jobwise since 2001 (though that first entry on March 24 could best be used to point out the frequent tedium of an assistant professor's life, not Hello, world so much as I'm in the office on another Saturday, world). In its first few years of existence, I was one of the hundreds of active participants on the H-Net collection of humanities e-mail lists. According to Google Scholar this morning, my five most-cited publications include my first book, two articles in standard journals (one sponsored by a scholarly society, one stand-alone), and two articles in an online-only journal. The most cited? My first article in an online journal. My education and professional life has been touched in fundamental ways by previous efforts to "hack the academy."

So what does this history mean? Some would say such incremental change is insufficient, and we should blow up higher education. In the Twitter stream tied to THATCamp this year, Mark Sample argued, "[H]igher ed is terminally ill and hacking it only prolongs its stay in the hospice." Historians of education are familiar with this rhetoric of radical reform, though the literature on it generally focuses on K-12 rather than higher education. There are plenty of attempts to "blow up" higher education by creating institutions that veer off from the plurality of practices. Sometimes those survive on the margins, or die out, and sometimes they become a model (if not necessarily an ideal). Antioch College, New College of Florida, Evergreen State College, and UC Santa Cruz have their origins in such attempts. Two of those started life as public institutions, one became public after a funding crisis, and Antioch's future is still in doubt. Or, if you want to go back further in time, both Clark University and Johns Hopkins University were founded with the intent to stay away from undergraduate education, and instead Johns Hopkins became the model for the modern large university, including high-priced tuition for undergraduates.

As an historian of education, I'd caution my more utopian colleagues that both institutions and people have to pay bills regularly, and that being able to pay bills allows innovation to thrive and spread. The wonderful thing about the internet is that you can do all sorts of intellectual work with minimal infrastructure. The damned thing about the internet is that resources still matter, especially if you want to foster a community of practice. Life involves compromise, even for hackers. With that in mind, here is a digital handful of ideas for hacking the academy, starting with the necessary and institutionalist and moving to what I would like to think of as the more inspirational:

0. Tenure-track faculty at research universities need to demonstrate competence in conventional ways. The bad news (if you were hoping to gain tenure at a research-oriented place with an experimental form of scholarship): if you are an assistant professor at a place that requires scholarship for tenure, you have roughly five years to get stuff published in a recognized way, and that means ways that external reviewers will recognize. The corollary: if you're a graduate student wanting to be a faculty member at a place that values research, you need to develop those competencies. The bittersweet news: because so few faculty are tenure-track at research universities, that means the vast majority of scholars can be innovative. That includes tenured faculty but also librarians, museum staff, and anyone who can find an alternative academic career path. If you are on the tenure track, you need to think about a career that lasts 20-30 years. If you demonstrate your chops in conventional ways in your first 5 years, you have the vast majority of your career to take greater risks.

1. We can build a broader coalition for reforming promotion considerations (but probably not tenure criteria) by discussing the value of taking risk in scholarship. If you're an aspiring digital humanist and are frustrated that a curated online website is not valued as scholarship in the same way as a university-press monograph, even if it's used by hundreds of classes or scholars worldwide, look at your colleagues who are conducting engaged scholarship in communities, where projects take years to get funded, help communities, and become translated into refereed articles. Or look at your colleagues who worked their tails off to earn tenure to find themselves as associate professors caring simultaneously for children and aging parents. Yes, the vast majority of them are women, and they find themselves with tenure but also with a gap in their scholarship record. The best way out of all these dilemmas is to argue that institutions should value long-term, risky projects when they demonstrate their value to the broader scholarly community. One could argue that the obligation of a scholar with tenure is not to continue to do the same work you did to earn tenure but to take greater intellectual risks. Let's find common cause by appealing to broader values.

2. The transition to post-publication review is in process. arXiv is leading the way as a recognized outlet for working papers in an entire discipline, and somehow physicists don't agonize about the peer-review process as journal publication still conveys an imprimatur of quality. How post-publication review develops is something I cannot predict, but there are a number of reasons why we are likely to head in a different direction, from the expenses of humanities journals to the diversification of bibliometrics and the weak-ethics of author-fee journals with high acceptance rates, or what some hard-sciences faculty refer to as "write-only" journals. Assistant professors may not be happy that a provost wants to see their h-index, but you should be happy that Google Scholar will find a good chunk of (if not everyone) who cited your conference paper from 3 years ago.

3. Senior scholars have an obligation to advocate for the ideas explained above. I expect to be around for another few decades, and I want my university to be a place where I like to work. What's the value of being a tenured full professor if we don't help colleagues and encourage risk-taking in scholarship? This involves both the realistic advice we have to give new scholars and ways to nudge academic administrators with arguments we know are more likely to appeal to them. If we don't speak up, we let the most powerful and conventional win by default, and we fail in our obligation to make the "codes of power" (see Lisa Delpit) explicit and open.

4. Expect large universities to abandon good initiatives on a regular basis unless there are forceful incentives that inhibit double-crossing. In April 2010, Yale University stopped contributing to the Public Library of Science journal system, despite a symbiotic relationship (where Yale scholars have increasingly contributed to PLoS journals). Institutional support: great idea. But there was nothing to inhibit Yale's withdrawal apart from reputational risk. There's a reason why Elsevier is hated: they're very effective at rent-seeking. Don't become Elsevier, but if you run an innovative project, don't avoid or hate the time you spend thinking up how to diversify income. It'll keep your people employed when Yale kicks your project to the curb.

5. Reputational markets are the tip of the iceberg in academic economies, and expanding/creating new economies is one route to hacking the academy in both peer review and funding. The Berkeley Electronic Press system has a formalized credit system for authors and reviewers in the form of its A&R bank. In the twitter stream for THATCamp Prime 2010, Jo Guldi suggested a pledge-support system for creative scholarly initiatives. This payback collaboration is a viable, sustainable model in other environments; for example, one early-childhood intervention program in Tennessee relies on a reciprocal-obligation model for services, where parents in the program are obliged to pay back services by becoming volunteers after their children exits.

6. Tight networks should raise red flags. The network of self-labeled digital humanists comprise mostly white academics, library and museum staff, and independent scholars. That is broader than disciplinary societies in one sense but misses lots of people who might consider themselves digital humanists if exposed to the idea, including the growing population of people connected to cultural heritage sites. That omission is a missed opportunity to make tools and conversations more useful as well as make digital humanities more sustainable in the long term. There is a solid reason for departments and similar structures to exist inside an organization, but your good sense should prompt occasional trips outside your hallway. Periodically ask, Who is missing from the conversation?

7. Some projects are going to be ephemeral; either plan for obsolescence or plan for periodic rebuilding. Archivists remind us regularly that formats are not forever. The same is true for individual projects that require continuous maintenance, whether specific intellectual enterprises or the infrastructure (such as base code). The earliest online journals began as e-mail lists; those which survive are now on the web. H-Net has atrophied in part because it has never been through a recent complete rebuild, despite internal advocacy for such rebuilding.

8. Some projects should be ephemeral. This is not necessarily a bad thing: like a sand mandala of Buddhist monks, an ephemeral project can teach us much during its existence even while and perhaps because everyone involved knows it is time-limited. If you work on a project that will most likely burn brightly for eighteen months and no more, be happy and up-front about that fact. Make your fans miss the project when it's gone.

9. Not everyone will or should be on the bleeding edge. Especially for specific tools, there is some maturity threshold before a piece of technology becomes more broadly usable (if it ever does). For example, online conference software exists, and particularly adept scholars can put together a virtual conference if they are willing to invest more effort than a lot of people might. With some effort I could probably create a one-day workshop using Google Wave. But how many would participate? In a few years, there might be a package that is closer to turnkey status, and then virtual conferences will be more feasible because organizing them will require less effort for the infrastructure.

A. A critical mass of users enables not only a rapid change of practice but the breaking of barriers. The corollary of not everyone's being on the bleeding edge is that one needs to know when enough people have a technology to assume its availability and to push hard at barriers. For example, now is the time to push unwieldy scholarly organizations to negotiate members' wifi in conference hotel contracts. Twitter may not exist in a few years, but internet access for attendees will make conferences more useful for whatever exists, to connect people and enable more engagement than listening to 20-minute paper readings.

B. Students need rules made explicit, and these include the hidden rules of life and scholarship, especially when a faculty member is trying something new and risky for students as well as the teacher. By rules I mean, "Here's how you get stuff done with minimal pain." And also the meta-stuff: "Here's what I'm trying that's new, here's why and what I expect you to learn, and please tell me when I'm screwing up." The immediate corollary inside a university or college is love your librarians, for they will often teach students what you forget to. The second corollary is reveal the hidden secrets in bits and pieces. I have very long undergraduate syllabi, but I know the students who most need the information are least likely to read and remember everything, so I expect to repeat the same information at key points in a term. The head of the martial-arts center I attend regularly introduces corrections with, "Here's a black-belt secret..." Everyone loves to know secrets, especially students.

C. Surplus time is necessary for students to be creative and rigorous. The explanation is left as an exercise for the reader just before going to bed. If you're working too late tonight to be able to think as you brush your teeth, please reread the first sentence of this paragraph.

D. We make our teaching more effective if we can figure out how the class can seduce students. My first year at USF, I was hoarse halfway through each semester, and I decided I needed to take voice lessons if I wanted my career to last without ruining my voice. One of the most important concepts I learned was that every time I took a breath, it was a chance to start a beautiful phrase. Every group of students has at least one wonderful new scholar, and on the first day of the term, you have not yet bored them. That doesn't mean hacking our teaching should focus on entertainment. But it should make the experience irresistible.

E. Can you explain what you do to your neighbors, and have you invited them to look at your website and at the websites of people and projects you admire? Academic freedom means that you do not cater to political whims of the moment, but higher education should not throw away the enormous benefit of being perceived as a public good. Since so much of hacking the academy results in public work, that should be public in a broader sense of being known to the general public.

F. Make timeto dance. Everyone gets grumpy on occasion, but it's hard to sustain scholarship or creativity (and get others to support you!) if you're permanently grumpy. If you are no longer motivated by the joy or beauty of what you're doing, rediscover it or reinvent what you're doing until you discover a new source of joy.

May 20, 2010

Texas and reality

Despite what I promised a few hours ago, this entry is not about coeducation, but current events in Texas are pushing my thoughts away from the value of college, at least for now. The rolling disaster that is the lame-duck-infested Texas Board of Education is both agonizing and fascinating, or one step above the formerly-creationist Kansas Board of Education (when the majority was in favor of teaching creation myth as part of science). Reading and listening to the more conservative board members leads me to conclude tentatively that while they will not say so explicitly, they really would like a curriculum that is based on a providential understanding of historical cause: America (and Texas) is blessed, and history shows how God has favored us, especially when we have been Godly.

That desire for providential history in public schools is wrong for two reasons. First, public schools in the United States should not be teaching religion as truth. (Teaching about religious beliefs and organizations as an important part of history is different. Teaching about religious beliefs as part of the cultural background for literature, myth, etc. is likewise different and perfectly acceptable.) The majority of the Texas board obviously disagrees with my interpretation of the First Amendment, but there's a second reason to avoid providential explanations of history: it is incompatible with the type of historical argumentation that is professionally acceptable to historians.

There are all sorts of historical explanations, metanarrative structures, and assumptions about human nature that professional historians would find plausible or at least acceptable to discuss as part of historical writings. But history as practiced today is about human nature and observable events, not providential explanations. That's as true of historians who have deeply-held religious views as it is for nonbelievers who write history. We just don't write deus ex machina history.

I know: we've been down this road before with debates over creationism and its close cousins: evolutionary biology is not a religion, and neither is standard history. But there's something that we can learn by thinking about history rather than science: the type of incommensurable perspectives that exist in the evolution/creationism divide is not there just because we're talking about fossils rather than human beings. That's close to the type of distinction that some refer to as mind-independent vs. mind-dependent phenomena. And I understand the appeal of that distinction.

But I have a different way of looking at the detritus of poststructuralism, and perhaps it's because I knew in writing Creating the Dropout that the bit about the construction of dropping out was sloppy in terms of handling the idea of social construction. I was focused on writing the story as detailed as was appropriate in an historical sense: when did "dropout" become the dominant term for adults who didn't have a high school diploma, what was the description that became associated with that, what did the choices at the time foreclose, etc. But as an historian who is generally more focused on the details than the meta-meta-level assumptions, I didn't do much more in talking about the construction of social problems than wave at Hilgartner and Bosk and go about my work. Did I mean that the stereotype associated with the term dropout was one of those paralinguistic structures that foreclosed alternatives, or that would spread and become an overturned irony over time? Was it part of a growing hegemony about the value of education? I apologize to anyone who was disappointed, but I was not up to the meta-para-hypertheoretical work that might have been involved. And no one really called me on that gap: reviewers generally acknowledged the story in the first few chapters and poked holes (some real and some virtual) in other pieces.

That doesn't mean that I am unread in relevant literature. I took my first-year proseminar with Lynn Hunt, and she walked us through Foucault, White, and a number of others who fall in the poststructuralist/deconstructionist canon (irony intended). But the question of whether language in the abstract performs the type of cultural work that some attribute to it paled in comparison with what people actually said about high school attrition in the 1950s, 1960s, and since. Given what Lynn's written since that year in criticizing the extreme forms of historiographical deconstruction, I think I may have made the right choice in how to spend my time, at least when it came to my first major research project.

But there is a larger question here of how to handle the fuzzy and malleable categorizations of (what we think of as) reality. Do we make choices about how to frame reality? Yes, of course, but in a relatively mundane sense of having to make some choice in how we investigate or describe the world. We can't avoid that choice, and for the moment I'll be agnostic on whether investigation is with scientific instruments or textual analysis (or something else), or whether communication is with language, mathematical symbols, or whatnot. Once the choice is made, that creates some structure about how we view reality, and it imposes at least a minimal cost on looking at things in a different way.

At this mundane forced-choice level, I'm essentially arguing that intellectual work is like the policy options for a country choosing whether you drive on the right or left side of the road. If you want most people to get anywhere on the road quickly and safely, you have to make a choice. We can debate whether the choice is political, economic, rational, irrational, etc., but a choice has to be made to get both quickly and safely, and there are consequences that flow from the choice, including signage, standard car equipment, and so forth. Note that this analogy doesn't touch issues such as correspondence with any underlying reality: It would be silly to claim that the choice of left or right has correspondence to Reality or Truth.

Instead, let me focus on the question of whether the choice at one time for the convention of driving on the left forecloses changing the convention, and what's required for such a change. At one level, the choice is mutually exclusive: a country cannot pick both rules and expect anything other than carnage when people drive faster than 5 mph. But at another level, the choice is resource-dependent: it's possible for England to change its rule so everyone drives on the right. It just would be a royal pain in the tuches.

So you can measure the rigidity of a convention in one sense by asking how expensive it would be to change it. Changing the side of a road for driving is expensive but possible. But you could imagine setting a rule that is impossible to change in the defined context. Unless you are driving on the Autobahn, most jurisdictions limit your speed to under the escape velocity of the planet. I don't think we could reverse that and require people to drive on the surface of this planet at greater than 7 miles per second.

Let's move away from driving conventions and back to how we talk about the universe. In both physical sciences and humanities, there are ways of classifying our fields that are nonexclusive and can be mixed; there are categories and ways of describing objects of interest that are exclusive but that can be switched from one to the other with some cost (i.e., exclusive but resource-dependent intellectual choices); and there are some choices that cannot be changed within that context (i.e., exclusive choices that you can't undo in the context you've created). Race, class, gender, disability, national origin, politics, language, etc., are all classification systems that can be mixed in the same context. No big deal there: we may choose to define categories of interest in different ways, but even if you call your categorization by the term class, and I call my different categorization class, we can just say they're different notions of class (or, as Ira Katznelson says in City Trenches, different layers of class). For the mathematically or notationally inclined, we could even index them as Class1, Class2, Class3, etc.

As I wrote at the top of this entry, I think there are exclusive choices that you can't undo in a specific context. If you're an academic historian, your arguments are going to eschew providential explanations of events. You can't undo that and still be in the field of history as I understand it. Regardless of whether the surface disagreements between me and some Texas education board members appear to be political or pedagogical or something else, I think the deep difference is that a number of them truly think public schooling should be teaching providential history or the "intelligent history design" equivalent (i.e., papered over). Again, that does not mean that historians or history teachers have to be agnostic or atheist, just that what they write or teach as historians isn't providential. (My high school history teacher Mr. Knowlton was one such person, a conservative evangelical who taught American history using primary sources and definitely non-providential arguments, though I know from conversations with him outside class that he clearly had providential beliefs outside his professional role.)

What I haven't talked about are examples in history (or other disciplines) of the exclusive but resource-dependent ways of categorizing reality. I'd be tempted to draw from physics (designing experiments to observe electrons as either particles or waves, but not both at the same time), but that's cheap. I will admit that it is late, I am tired, this entry is long as is, and maybe leaving this open-ended will draw interesting comments or enough suspense to keep you reading my blog. But please chime in on comments: am I all wet, on track, and can I be both at the same time without the universe exploding?

The value of college I

Over the past week, I had been collecting a number of references to recent online discussions of the value of education when the New York Times column by Jacques Steinberg highlighting the views of Richard Vedder and Charles Murray appeared. Claus von Zastrow (among others) has already pointed out that given the fact that the advocates of the "you don't need college" position are highly educated, this reads as an argument that other people's children shouldn't go to college. I sometimes have a bit of fun when Bill Gates talks about the importance of college--"do as I say, not as I do"--but Gates errs on the sides of generosity in terms of what he'd like others to accomplish. Not so Vedder or Murray.

I'm going back over Goldin and Katz's The Race between Education and Technology with a finer-toothed comb than when it first came out in 2008, and I'll probably write a number of posts on this topic. Generally, the literature on the value of higher education (or formal schooling more broadly) is not particularly nuanced. It's human capital and a boost to income!No, it's a queueing process!No, it's a confirmation of inherited intelligence! It's a floor wax! It's a dessert topping!

I'll start with an historical perspective, a warning about loose generalizations: let's stop talking about "higher education" in the abstract, as if it's the output of a utility. Colleges and universities are specific institutions, and the value that students receive from them are dependent on context. In the nineteenth century, a number of states and some cities created normal schools, or schools designed to train teachers. But as Chris Ogren and others have pointed out, in addition to the teacher-training function, public normal schools were often the nearest place where anyone could get something beyond rudimentary schooling, so they inevitably became general schools. When normal schools became teachers colleges, you saw the same phenomenon; Lyndon Johnson attended a teachers college because that was where he could go to college, period. To see the history of normal schools and teachers colleges entirely through the lens of teacher education would be historically inaccurate and narrow.

Or, to take another example, the history of vocational education is not just about narrow trade schooling and the denial of educational opportunity through tracking. That's a large part of it (e.g., ), but again, context is everything. Ira Katznelson and Margaret Weir point out the disputes between labor and business in the structure of vocational education in Chicago. And both Kathryn Neckerman and Bill Graebner have pointed out that in many northern cities, vocational-technical high schools were one cut above comprehensive high schools, sufficiently so that working-class whites fought to keep African American students out of them. In my archival research for my dissertation and first book, I saw something similar in the Atlanta-area vo-tech school in the early 1960s, where administrators fought to prevent it from being what they perceived might be the dumping ground for area schools. Again, institutional context is important.

The reality is that higher education serves several "functions." Some of that could be considered human capital, but the only way to call all of the value human capital is to make the term meaningless. And plenty of higher (and other) education also helps advantaged families hoard those advantages, but it's far from a hermetic process and far less tilted towards the wealthy than plenty of other areas of life (housing, the labor market, tax codes, health-care access, etc.). And all of this type of analysis is predicated on the ability to identify predictable consequences of education. But that's not true, and the prime example is the coeducation of primary education in 19th century America. That's the topic for the next entry on the value of education.

May 18, 2010

Crabby comments from a higher-ed union activist

I'm in a slightly cranky mood from spending most of the last two days' work hours on copyediting. This says little about the material I was copyediting and more about the nature of the task (and why I did not get a job to spend all of my time copyediting). So I'm ending the day in a crabby mood and still want to be productive, but I may not want to contact any people about current issues. So it's time for crabby comments about situations that popped up in the semi-distant past (at least 12 months ago), and I'll refer to them vaguely enough that they could apply to all sorts of situations. If you think any of the following is about you, you're probably wrong.

Did you realize that if you hadn't tried the procedural short-cut, you would have won the argument on the merits?

You can probably get away with half of the stuff you're loading on your faculty. Which half do you care about?

The union has membership, staff, institutional memory, and access to lawyers. We're plum out of magic wands.

I know you've got a Ph.D. and a winning smile, but other people can remember things, too, and sometimes we check factual claims.

Your hallway is not the whole university.

I don't care what Gordon's or Drew's trustees let them do.

You've got several very smart administrators in your office. We've got several hundred members. Maybe you could outthink the lot of us, but your behavior makes me suspect you've never played a role-playing game.

If wishes were fishes and gripes were wet wipes... no, let's not go there.

Yes, you've got academic freedom and I'll defend that to the hilt, but it may not be wise for assistant professors to run up a large debt in deviance credits. The interest alone is murder.

You didn't take advantage of the opportunity when it was in your lap, and you're now hoping the expired opportunity returns and doubles or triples in size. On your way out, could you please pick up my jaw and hand it back to me?

I refuse to answer on the grounds that it may incriminate you.

Your strategic vision appears to rely on throwing everything against the wall and seeing what sticks. Pardon my brashness, but not even the most tendentious Soviet/corporate five-year plan has had "al dente" as a bullet point.

I see you're trying out the new Elizabeth Taylor fragrance, "Entitlement." I may be overstepping the bounds of friendship by saying this, but I think you've applied a bit too much.

I'll give you 10 out of 10 for venomous intent, but my long-term exposure to teenagers has immunized me.

I'm sure I can take this horrible day and turn it into a conference presentation or article.

It's stronger technically than a previous paper by Chingos and West that I commented on recently. As a former editor, I'd guess it's one thorough scrub away from being publishable at a reasonably selective journal.

It's based on a clever concept, comparing districts and schools that had "far to go" on class size with districts and schools that were very close to the starting mandates and framing the comparison as one of "if you give districts and schools more money, do you get more bang from your buck by mandating that they use that money to lower class size?" That's not the question on the ballot this fall, but it's a reasonable slice given the difficulties of disentangling policy effects. And if you know the "Monday's Child" rhyme, you'll understand why I'm calling this a Thursday's-child research design.

The paper focuses on grades 4-8, which is an important contribution to the literature because of the focus of much class-size research on primary grades. But that focus comes because grades 4-8 was the policy interval with the most variation in starting points among districts, and so the paper can't tell us about class-size effects at other grades.

Chingos addressed a number of potential weaknesses in reasonable fashion. There are two weaknesses remaining that struck me on first reading the paper -- the failure to address student migration (both interstate and intrastate) and the failure to consider "contamination" of the design by student experiences with primary-grade class-size reduction. The first can be addressed by some alternate samples, or even identification of students who moved from "control" districts to Thursday's-child districts. But the second is tough to address: while the majority of districts didn't have far to go on grades 4-8 starting in 2004, most of the "control" (or not-far-to-go) districts for grades 4-8 were still "far to go" districts as far as primary grades were concerned, and for intermediate-grade students in the third and later years, their test scores would have reflected experiences with class-size reduction in primary grades.

There are some interesting variations in tables and effects by specific grade.

As I told reporter Ron Matus yesterday, I'm sure this will be spun out of all proportion to the study because there's little solid research on Florida's class-size mandate, the stakes with the ballot measure to change the mandate are high, and the proponents of changing the mandate are evidently starting well behind the eight-ball with public opinion. So the temptation to exaggerate the findings is high. Jeff Henig would not be surprised.. and speaking of which, congrats to Henig for the AERA award for Spin Cycle! It's highly deserved.

May 2, 2010

The theater of basing a majority of evaluation on test scores

Now that SB 6 is dead, that a governor's task force on RTTT came to a compromise in a single day, and it looks like there is some direction for teacher evaluation in Florida that's acceptable to Florida's K-12 teachers unions, it's time to take stock of the rhetorical stance SB 6 supporters had that a "majority" of a teacher's evaluation had to depend on student test scores. I've seen this pop up in other states, so it's a common rhetorical stance. Let's get a few things off the table first: this is not based on any research, and the supporters have no clearer idea of what "majority of a teacher's evaluation" might mean than supporters of the "65% solution" had any clue what spending money in a classroom meant. For that matter, neither did I as a skeptic (about either proposal).

So the "majority on test scores" stance is political, then. That's fine as a minimal statement; almost all decisions about pay structures are political in a broad sense rather than based on research, and to some extent they're reactive. Teacher pay scales became standardized to protect bureaucratic structures from (and sometimes in response to) accusations of corruption, and the single salary schedule is a response historically to gross pay inequity.

I'll go further: I don't think there's a way to avoid political values embedded in pay structures. Once you involve public money and a service most people connect with citizenship (education), you've got politics, however well structured and justified by reference to neutral statements of organizational need. On that level, performance pay is justifiable from the sense of satisfying public perceptions about how teachers should be paid. That was explicit in Denver's ProComp plan: the voters approved higher taxes in return for a performance-pay structure.

The problem with the "majority based on test score" position is twofold. One is the obvious one: it's divisive, and many parents and other community members are offended by the idea. Here, Diane Ravitch spoke for millions when she criticized SB 6. But there's another problem: it obscures the evaluation process rather than clarifying it. By reference to an implied point-based system, it fails to focus on what matters in a teacher evaluation system in terms of either an algorithm or underlying concepts.

I've written a bit about point-based systems, and because the focus of my paper was elsewhere, I didn't have a chance to talk about the limit of point-based scoring systems: it matters not where you can earn points but where you might lose points. I learned this in high school when I was a debater: individual raters have an implied comfortable range for scores, and it's the range of scores that matters, not the total number of points available in different categories. If raters have different effective ceilings as well as ranges (i.e., it is impossible for people to earn perfect scores with some raters, while others commonly hand out full marks), then the raters with the largest ranges of scores exert more power over final results than raters who have a very narrow range.

Similarly, components of any point-based system will have differential impact on final results when they have broader ranges in practice regardless of the proportion of the scale that derives from individual components. Imagine a teacher evaluation system with 100 points. Suppose 60 points comes from student test scores, and the range is restricted for most teachers to between 52 and 60 points. On the other hand, suppose 30 points in this hypothetical evaluation comes from direct observation, and the range of scores is between 10 and 30 (and more than a handful of teachers may earn the low score). Which component has the greatest influence on final results? It's the 30-point direct-observation component in this thought experiment, because in this hypothetical example teachers can lose more than twice the number of points there than through student test scores.

But the "majority of evaluation" rhetoric does more than obscure the real power in point-based systems: it obscures the question of what teachers are responsible for. "Outcomes!" says the supporter. Right, I say: that doesn't say a darned thing about the types of outcomes that will make the difference in evaluation. In Florida, Louisiana, and other states where people have pushed a majority from test scores approach, the push has been to create a mandate and defer the implementation to a regulatory process. That's a nice illusionist's trick if you can get away with it, but the process of implementation always mediates absolutist mandates, and then the legislature is giving up what mediates the test scores.

There are three ways I can see that test scores' impact on evaluations would be mediated in any system (and yes, I'm including SB 6 here): ad hoc (i.e., caprice), by reference to student disadvantage (i.e., blame-shifting), or by reference to teacher behaviors in classrooms (i.e., standards of practice). Without any legislative guidance, ad hoc and capricious mediation is likely (probably by the temperament and philosophy of the administrator with the greatest authority over evaluation). More destructive than ad hoc mediation would be blame-shifting: a teacher would be held blameless if someone else/something else (poverty, language, presumed parental neglect, etc.) can be blamed instead. Bad, bad idea.

Of the three options that come to mind tonight, mediating test scores by professional standards of practice seems the most productive. But then that raises the central question: if the use of test scores is inevitably subject to mediation, and the best choice for that mediation is through professional standards of practice, why not base evaluation on professional standards of practice to begin with--for example, to let an evaluation that documents effective practice create the rebuttal presumption of effectiveness?

The answer here is two-fold: one is that there is no agreed-upon standards of practice for teaching more generally, other than by crude and obvious standards (don't beat your students) or by reference to effects (keep your students' attention). The other explanation is that even if there were agreed-upon standards of practice, the process would be sufficiently messy as to irritate the sensibilities of those who advocate the putatively cleaner "majority from test score" approach.

The result is that instead of getting a messy but constructive system based on developing standards of practice, any such system that putatively bases the majority of a teacher's evaluation on test score is going to get ad hoc or blame-shifting mediation through the back door.

Update: Linda Perlstein noticed the 50% rhetoric and should get credit for the pattern recognition. Consultants' advice? Hmmn... looks like an interlocking-directorate phenomenon (no conspiracy needed).