Monday, January 31, 2011

Like many people we’ve been glued to the news unfolding in Egypt and thinking of what we could do to help people on the ground. Over the weekend we came up with the idea of a speak-to-tweet service—the ability for anyone to tweet using just a voice connection.

We worked with a small team of engineers from Twitter, Google and SayNow, a company we acquired last week, to make this idea a reality. It’s already live and anyone can tweet by simply leaving a voicemail on one of these international phone numbers (+16504194196 or +390662207294 or +97316199855) and the service will instantly tweet the message using the hashtag #egypt. No Internet connection is required. People can listen to the messages by dialing the same phone numbers or going to twitter.com/speak2tweet.

We hope that this will go some way to helping people in Egypt stay connected at this very difficult time. Our thoughts are with everyone there.

100% Renewable Energy in 40 Years Not Limited to Our Wildest Dreams: Study

New research suggests the whole world could switch to renewable energy sources using current tech in just 20 to 40 years. It would cost no more than current energy, and would have big economic and eco payoffs. The only barriers are down to social, business, and political inertia.

We all know about renewable energy--it's been around for years, and is key to solving the global warming (and end-of-oil) crisis. Nowadays it's good to be green, and research into the millions of different aspects of the tech is skyrocketing. But a Stanford research team has just compiled an innovative, lateral-thinking study that says even using current available technology the entire world could switch 100% of its energy needs to renewable sources in just a handful of decades. How is this possible?

Current tech is good enough

The research from Mark Z. Jacobson and team involves making all new energy production plants use renewable energy by 2030, and then converting older existing plants by 2050. In the new world order, almost everything would run off electricity. Ninety percent of the production would come from windmills and solarenergyplants (already very well established technologies) and the remaining 10% would come from hydroelectric power, geothermal, and wave/tidal power. Mobile things--cars, trains, ships and such--would run on hydrogen-powered fuel cells, and aircraft would burn hydrogenfuel. The hydrogen itself would come from green-electric generation processes.

All of this plan requires no more than a dedicated push to exploit existing technology and to network it all together in an intelligent way--because demand varies from place to place, throughout the day, and as seasons change, and the sun, wind, and waves don't necessarily give power all the time, everywhere. "If you combine them as one commodity and use hydroelectric to fill in gaps [as it's a reliable battery-like resource], it is a lot easier to match demand," Jacobson notes. A supergrid, with long-distance links, international cooperation and really smart energy management is needed. (Good job we're alreadybuilding one).

Will it cost more?

Nope. Making the changes will take time, effort and money--because you have to build a lot of new equipment, and link up power grids across the world. Spinning up green-power industries to build devices at a global scale will also cost money, as will winding down and deconstructing the infrastructure in place to support coal, oil, gas and even nuclear electricity generation.

But "when you actually account for all the costs to society--including medical costs--of the current fuel structure, the costs of our plan are relatively similar to what we have today," according to Jacobson. That medical reference is to the health benefits of reducing pollution on a global scale, as well as side-effects like deaths from warming-induced natural disasters.

Will it cost more in the long run?

Nope, it may cost less. Due to the incontrovertible laws of thermodynamics and other bits of physics, "heat engines" like the non-renewable power stations and car engines we run today are way less energy efficient than an all-electric process. The Stanford plan suggests global energy needs would drop by 30% due to this efficiency boost, meaning we'd actually need less power--and if the business models evolve to support this norm, individuals may pay less for their energy.

Won't we pepper the Earth with windmills and solar farms and hydroelectric dams?

Nope, Stanford's plan would require 0.4% of the world's land (mainly for solar power) and the spacing between windmills accounts for another 0.6%--although you can use this area for farming and catering for other needs. One percent of the windmills are already in place, and Jacobson notes "the actual footprint required by wind turbines to power half the world's energy is less than the area of Manhattan."

Considering how much space is taken up by power stations and coal mines--facilities that would be closed in the plan--this isn't too much of a sacrifice. And a significant share of the wind farms could be offshore, to satisfy NIMBYism.

Why don't we do it then?

Inertia. We're all used to the current way of things, and rethinking everything from how your car works to looking at a landscape where power windmills go from rare to the norm involves a big effort--a "large scale transformation" on a global scale. Governments are notoriously slow-footed when it comes to this sort of change, and the Stanford plan involves so many innovations and international cooperation that the complexity is almost beyond imagination.

Existing businesses who rely on coal, oil, gas (and their byproducts, like the airline industry's need for aviation fuel) will be reluctant too.

But we've done similar things, as Jaconson notes--it's an effort "comparable to the Apollo moon project or constructing the interstate highway system," just compressed into a short timescale and requiring action from a majority of nations.

EFF has uncovered widespread violations stemming from FBI intelligence investigations from 2001 - 2008. In a report released today, EFF documents alarming trends in the Bureau’s intelligence investigation practices, suggesting that FBI intelligence investigations have compromised the civil liberties of American citizens far more frequently, and to a greater extent, than was previously assumed.

• Evidence of delays of 2.5 years, on average, between the occurrence of a violation and its eventual reporting to the Intelligence Oversight Board

• Reports of serious misconduct by FBI agents including lying in declarations to courts, using improper evidence to obtain grand jury subpoenas, and accessing password-protected files without a warrant

• Indications that the FBI may have committed upwards of 40,000 possible intelligence violations in the 9 years since 9/11

EFF's report stems from analysis of nearly 2,500 pages of FBI documents, consisting of reports of FBI intelligence violations made to the Intelligence Oversight Board — an independent, civilian intelligence-monitoring board that reports to the President on the legality of foreign and domestic intelligence operations. The documents constitute the most complete picture of post-9/11 FBI intelligence abuses available to the public. Our earlier analysis of the documents showed the FBI's arbitrary disclosure practices.

EFF's report underscores the need for greater transparency and oversight in the intelligence community. As part of our ongoing effort to inform the public and elected officials about abusive intelligence investigations, we are distributing copies of the report to members of Congress.

The late Permian extinction, which kicked off roughly 250 million years ago, has a rather gruesome nickname: the Great Dying. Over 90 percent of the species in the oceans went extinct in the geological blink of an eye, and similar devastation took place on land. It's about as close as we've come to having multicellular life wiped out. The timing of the event coincides with a volcanic outburst that covered an area the size of Western Europe in volcanic rock. That might be enough to trigger a major catastrophe on its own, but new research indicates that the hot magma ignited coal deposits, sending toxic coal ash into the oceans.

The remains of these eruptions, called the Siberian Traps, now cover about 2 million square kilometers of Russia. The rock formation is what's called a flood basalt, thought to be caused by a plume of hot mantle breaking through to the surface. The Siberian Traps may be the largest event of this sort we know about, and the dimensions are staggering: over 1,000 Gt (Gigatonnes) of magma were released during the eruptions that created them, and they are thought to have put material into a plume that rose over 40 kilometers into the atmosphere.

The effects on life were devastating. Massive ash falls, huge changes in the carbon cycle, ocean acidification, and climate change all accompanied the eruptions, and all of those are capable of pushing species to extinction. But is that really enough to account for wiping out over 90 percent of the life in the oceans?

Some researchers don't think so, and have focused on a secondary effect of the eruptions: burning coal. There is evidence that the hot magma intruded into large deposits of coal found in Siberia and set it alight. Some estimates suggest that over 3 trillion tons of carbon could have been placed into the atmosphere through the burning of coal alone (that's in addition to the carbon dioxide released by the volcanism proper). That release would come in the form of methane, a potent greenhouse gas. Methane is rapidly oxidized into carbon dioxide, which could then contribute to ocean acidification.

As if all of that weren't enough, the new paper, published in Nature Geoescience, indicates that the Siberian Traps eruptions might have added another insult to the oceans: toxic coal ash. The authors examined deep ocean sediments from a site that was off the west coast of the supercontinent at the time. To get there with the prevailing winds, material from the eruption would have to travel around the globe, a distance the authors estimate as more than 20,000 kilometers. And yet the sediments contain organic material that, under the microscope, looks remarkably similar to coal ash obtained from a modern power plant.

The authors were able to detect three pulses of this material derived from coal burning in the half-million years before the onset of the Great Dying, with the third and most significant ending just as marine life collapsed. Each of them were associated with changes in the carbon cycle, either resulting from the large release of the eruptions themselves, or the burning of organic materials that continued in their wake.

This combustion material is known to stress aquatic ecosystems in two ways. To begin with, the coal ash will block enough sunlight to inhibit photosynthesis, a major source of the ocean's dissolved oxygen. That creates anoxic conditions, and isotope ratios confirm that the oceans probably experienced an anoxia event that coincided with the last major eruption. In addition, coal ash carries toxic metals with it; levels of chromium increased with each of the eruptions, and peaked with the third and final one. Thus, the coal ash itself probably contributed directly to the conditions that were so harmful to life in the oceans.

It's clear from this data that the formation of the Siberian Traps created conditions that would severely stress life in the oceans through a variety of mechanisms. Although problems would obviously have started before the Great Dying itself, the final, critical event seems to have been the most severe. The preponderance of evidence clearly links the extinction to the eruptions that made Traps themselves.

The biggest open question is whether all the factors were global in nature. Diffusion of gasses is rapid, so issues like ocean acidification would clearly have a global impact. But the spread of fly ash is very dependent on the prevailing winds, and the site examined by the authors is at roughly the same latitude as the eruptions themselves. Sampling of sites further removed from the Siberian Traps may indicate whether what the authors see there (which they refer to as "Catastrophic dispersion of coal fly ash" in the title) was a truly global catastrophe.

If Camden, New Jersey, becomes the first American community to lower its medical costs, it will have a murder to thank. At nine-fifty on a February night in 2001, a twenty-two-year-old black man was shot while driving his Ford Taurus station wagon through a neighborhood on the edge of the Rutgers University campus. The victim lay motionless in the street beside the open door on the driver’s side, as if the car had ejected him. A neighborhood couple, a physical therapist and a volunteer firefighter, approached to see if they could help, but police waved them back.

“He’s not going to make it,” an officer reportedly told the physical therapist. “He’s pretty much dead.” She called a physician, Jeffrey Brenner, who lived a few doors up the street, and he ran to the scene with a stethoscope and a pocket ventilation mask. After some discussion, the police let him enter the crime scene and attend to the victim. Witnesses told the local newspaper that he was the first person to lay hands on the man.

“He was slightly overweight, turned on his side,” Brenner recalls. There was glass everywhere. Although the victim had been shot several times and many minutes had passed, his body felt warm. Brenner checked his neck for a carotid pulse. The man was alive. Brenner began the chest compressions and rescue breathing that should have been started long before. But the young man, who turned out to be a Rutgers student, died soon afterward.

The incident became a local scandal. The student’s injuries may not have been survivable, but the police couldn’t have known that. After the ambulance came, Brenner confronted one of the officers to ask why they hadn’t tried to rescue him.

“We didn’t want to dislodge the bullet,” he recalls the policeman saying. It was a ridiculous answer, a brushoff, and Brenner couldn’t let it go.

He was thirty-one years old at the time, a skinny, thick-bearded, soft-spoken family physician who had grown up in a bedroom suburb of Philadelphia. As a medical student at Robert Wood Johnson Medical School, in Piscataway, he had planned to become a neuroscientist. But he volunteered once a week in a free primary-care clinic for poor immigrants, and he found the work there more challenging than anything he was doing in the laboratory. The guy studying neuronal stem cells soon became the guy studying Spanish and training to become one of the few family physicians in his class. Once he completed his residency, in 1998, he joined the staff of a family-medicine practice in Camden. It was in a cheaply constructed, boxlike, one-story building on a desolate street of bars, car-repair shops, and empty lots. But he was young and eager to recapture the sense of purpose he’d felt volunteering at the clinic during medical school.

Few people shared his sense of possibility. Camden was in civic free fall, on its way to becoming one of the poorest, most crime-ridden cities in the nation. The local school system had gone into receivership. Corruption and mismanagement soon prompted a state takeover of the entire city. Just getting the sewage system to work could be a problem. The neglect of this anonymous shooting victim on Brenner’s street was another instance of a city that had given up, and Brenner was tired of wondering why it had to be that way.

Around that time, a police reform commission was created, and Brenner was asked to serve as one of its two citizen members. He agreed and, to his surprise, became completely absorbed. The experts they called in explained the basic principles of effective community policing. He learned about George Kelling and James Q. Wilson’s “broken-windows” theory, which argued that minor, visible neighborhood disorder breeds major crime. He learned about the former New York City police commissioner William Bratton and the Compstat approach to policing that he had championed in the nineties, which centered on mapping crime and focussing resources on the hot spots. The reform panel pushed the Camden Police Department to create computerized crime maps, and to change police beats and shifts to focus on the worst areas and times.

When the police wouldn’t make the crime maps, Brenner made his own. He persuaded Camden’s three main hospitals to let him have access to their medical billing records. He transferred the reams of data files onto a desktop computer, spent weeks figuring out how to pull the chaos of information into a searchable database, and then started tabulating the emergency-room visits of victims of serious assault. He created maps showing where the crime victims lived. He pushed for policies that would let the Camden police chief assign shifts based on the crime statistics—only to find himself in a showdown with the police unions.

“He has no clue,” the president of the city police superiors’ union said to the Philadelphia Inquirer. “I just think that his comments about what kind of schedule we should be on, how we should be deployed, are laughable.”

The unions kept the provisions out of the contract. The reform commission disbanded; Brenner withdrew from the cause, beaten. But he continued to dig into the database on his computer, now mostly out of idle interest.

Besides looking at assault patterns, he began studying patterns in the way patients flowed into and out of Camden’s hospitals. “I’d just sit there and play with the data for hours,” he says, and the more he played the more he found. For instance, he ran the data on the locations where ambulances picked up patients with fall injuries, and discovered that a single building in central Camden sent more people to the hospital with serious falls—fifty-seven elderly in two years—than any other in the city, resulting in almost three million dollars in health-care bills. “It was just this amazing window into the health-care delivery system,” he says.

So he took what he learned from police reform and tried a Compstat approach to the city’s health-care performance—a Healthstat, so to speak. He made block-by-block maps of the city, color-coded by the hospital costs of its residents, and looked for the hot spots. The two most expensive city blocks were in north Camden, one that had a large nursing home called Abigail House and one that had a low-income housing tower called Northgate II. He found that between January of 2002 and June of 2008 some nine hundred people in the two buildings accounted for more than four thousand hospital visits and about two hundred million dollars in health-care bills. One patient had three hundred and twenty-four admissions in five years. The most expensive patient cost insurers $3.5 million.

Brenner wasn’t all that interested in costs; he was more interested in helping people who received bad health care. But in his experience the people with the highest medical costs—the people cycling in and out of the hospital—were usually the people receiving the worst care. “Emergency-room visits and hospital admissions should be considered failures of the health-care system until proven otherwise,” he told me—failures of prevention and of timely, effective care.

If he could find the people whose use of medical care was highest, he figured, he could do something to help them. If he helped them, he would also be lowering their health-care costs. And, if the stats approach to crime was right, targeting those with the highest health-care costs would help lower the entire city’s health-care costs. His calculations revealed that just one per cent of the hundred thousand people who made use of Camden’s medical facilities accounted for thirty per cent of its costs. That’s only a thousand people—about half the size of a typical family physician’s panel of patients.

Things, of course, got complicated. It would have taken months to get the approvals needed to pull names out of the data and approach people, and he was impatient to get started. So, in the spring of 2007, he held a meeting with a few social workers and emergency-room doctors from hospitals around the city. He showed them the cost statistics and use patterns of the most expensive one per cent. “These are the people I want to help you with,” he said. He asked for assistance reaching them. “Introduce me to your worst-of-the-worst patients,” he said.

They did. Then he got permission to look up the patients’ data to confirm where they were on his cost map. “For all the stupid, expensive, predictive-modelling software that the big venders sell,” he says, “you just ask the doctors, ‘Who are your most difficult patients?,’ and they can identify them.”

The first person they found for him was a man in his mid-forties whom I’ll call Frank Hendricks. Hendricks had severe congestive heart failure, chronic asthma, uncontrolled diabetes, hypothyroidism, gout, and a history of smoking and alcohol abuse. He weighed five hundred and sixty pounds. In the previous three years, he had spent as much time in hospitals as out. When Brenner met him, he was in intensive care with a tracheotomy and a feeding tube, having developed septic shock from a gallbladder infection.

Brenner visited him daily. “I just basically sat in his room like I was a third-year med student, hanging out with him for an hour, hour and a half every day, trying to figure out what makes the guy tick,” he recalled. He learned that Hendricks used to be an auto detailer and a cook. He had a longtime girlfriend and two children, now grown. A toxic combination of poor health, Johnnie Walker Red, and, it emerged, cocaine addiction had left him unreliably employed, uninsured, and living in a welfare motel. He had no consistent set of doctors, and almost no prospects for turning his situation around.

After several months, he had recovered enough to be discharged. But, out in the world, his life was simply another hospitalization waiting to happen. By then, however, Brenner had figured out a few things he could do to help. Some of it was simple doctor stuff. He made sure he followed Hendricks closely enough to recognize when serious problems were emerging. He double-checked that the plans and prescriptions the specialists had made for Hendricks’s many problems actually fit together—and, when they didn’t, he got on the phone to sort things out. He teamed up with a nurse practitioner who could make home visits to check blood-sugar levels and blood pressure, teach Hendricks about what he could do to stay healthy, and make sure he was getting his medications.

A lot of what Brenner had to do, though, went beyond the usual doctor stuff. Brenner got a social worker to help Hendricks apply for disability insurance, so that he could leave the chaos of welfare motels, and have access to a consistent set of physicians. The team also pushed him to find sources of stability and value in his life. They got him to return to Alcoholics Anonymous, and, when Brenner found out that he was a devout Christian, he urged him to return to church. He told Hendricks that he needed to cook his own food once in a while, so he could get back in the habit of doing it. The main thing he was up against was Hendricks’s hopelessness. He’d given up. “Can you imagine being in the hospital that long, what that does to you?” Brenner asked.

I spoke to Hendricks recently. He has gone without alcohol for a year, cocaine for two years, and smoking for three years. He lives with his girlfriend in a safer neighborhood, goes to church, and weathers family crises. He cooks his own meals now. His diabetes and congestive heart failure are under much better control. He’s lost two hundred and twenty pounds, which means, among other things, that if he falls he can pick himself up, rather than having to call for an ambulance.

“The fun thing about this work is that you can be there when the light switch goes on for a patient,” Brenner told me. “It doesn’t happen at the pace we want. But you can see it happen.”

With Hendricks, there was no miraculous turnaround. “Working with him didn’t feel any different from working with any patient on smoking, bad diet, not exercising—working on any particular rut someone has gotten into,” Brenner said. “People are people, and they get into situations they don’t necessarily plan on. My philosophy about primary care is that the only person who has changed anyone’s life is their mother. The reason is that she cares about them, and she says the same simple thing over and over and over.” So he tries to care, and to say a few simple things over and over and over.

I asked Hendricks what he made of Brenner when they first met.

“He struck me as odd,” Hendricks said. “His appearance was not what I expected of a young, clean-cut doctor.” There was that beard. There was his manner, too. “His whole premise was ‘I’m here for you. I’m not here to be a part of the medical system. I’m here to get you back on your feet.’ ”

An ordinary cold can still be a major setback for Hendricks. He told me that he’d been in the hospital four times this past summer. But the stays were a few days at most, and he’s had no more cataclysmic, weeks-long I.C.U. stays.

Was this kind of success replicable? As word went out about Brenner’s interest in patients like Hendricks, he received more referrals. Camden doctors were delighted to have someone help with their “worst of the worst.” He took on half a dozen patients, then two dozen, then more. It became increasingly difficult to do this work alongside his regular medical practice. The clinic was already under financial strain, and received nothing for assisting these patients. If it were up to him, he’d recruit a whole staff of primary-care doctors and nurses and social workers, based right in the neighborhoods where the costliest patients lived. With the tens of millions of dollars in hospital bills they could save, he’d pay the staff double to serve as Camden’s élite medical force and to rescue the city’s health-care system.

But that’s not how the health-insurance system is built. So he applied for small grants from philanthropies like the Robert Wood Johnson Foundation and the Merck Foundation. The money allowed him to ramp up his data system and hire a few people, like the nurse practitioner and the social worker who had helped him with Hendricks. He had some desk space at Cooper Hospital, and he turned it over to what he named the Camden Coalition of Healthcare Providers. He spoke to people who had been doing similar work, studied “medical home” programs for the chronically ill in Seattle, San Francisco, and Pennsylvania, and adopted some of their lessons. By late 2010, his team had provided care for more than three hundred people on his “super-utilizer” map.

I spent a day with Kathy Jackson, the nurse practitioner, and Jessica Cordero, a medical assistant, to see what they did. The Camden Coalition doesn’t have enough money for a clinic where they can see patients. They rely exclusively on home visits and phone calls.

Over the phone, they inquire about emerging health issues, check for insurance or housing problems, ask about unfilled prescriptions. All the patients get the team’s urgent-call number, which is covered by someone who can help them through a health crisis. Usually, the issue can be resolved on the spot—it’s a headache or a cough or the like—but sometimes it requires an unplanned home visit, to perform an examination, order some tests, provide a prescription. Only occasionally does it require an emergency room.

Patients wouldn’t make the call in the first place if the person picking up weren’t someone like Jackson or Brenner—someone they already knew and trusted. Even so, patients can disappear for days or weeks at a time. “High-utilizer work is about building relationships with people who are in crisis,” Brenner said. “The ones you build a relationship with, you can change behavior. Half we can build a relationship with. Half we can’t.”

One patient I spent time with illustrated the challenges. If you were a doctor meeting him in your office, you would quickly figure out that his major problems were moderate developmental deficits and out-of-control hypertension and diabetes. His blood pressure and blood sugars were so high that, at the age of thirty-nine, he was already developing blindness and advanced kidney disease. Unless something changed, he was perhaps six months away from complete kidney failure.

You might decide to increase his insulin dose and change his blood-pressure medicine. But you wouldn’t grasp what the real problem was until you walked up the cracked concrete steps of the two-story brownstone where he lives with his mother, waited for him to shove aside the old newspapers and unopened mail blocking the door, noticed Cordero’s shake of the head warning you not to take the rumpled seat he’s offering because of the ant trail running across it, and took in the stack of dead computer monitors, the barking mutt chained to an inner doorway, and the rotten fruit on a newspaper-covered tabletop. According to a state evaluation, he was capable of handling his medications, and, besides, he lived with his mother, who could help. But one look made it clear that they were both incapable.

Jackson asked him whether he was taking his blood-pressure pills each day. Yes, he said. Could he show her the pill bottles? As it turned out, he hadn’t taken any pills since she’d last visited, the week before. His finger-stick blood sugar was twice the normal level. He needed a better living situation. The state had turned him down for placement in supervised housing, pointing to his test scores. But after months of paperwork—during which he steadily worsened, passing in and out of hospitals—the team was finally able to get him into housing where his medications could be dispensed on a schedule. He had made an overnight visit the previous weekend to test the place out.

“I liked it,” he said. He moved in the next week. And, with that, he got a chance to avert dialysis—and its tens of thousands of dollars in annual costs—at least for a while.

Not everyone lets the team members into his or her life. One of their patients is a young woman of no fixed address, with asthma and a crack-cocaine habit. The crack causes severe asthma attacks and puts her in the hospital over and over again. The team members have managed occasionally to track her down in emergency rooms or recognize her on street corners. All they can do is give her their number, and offer their help if she ever wanted it. She hasn’t.

Work like this has proved all-consuming. In May, 2009, Brenner closed his regular medical practice to focus on the program full time. It remains unclear how the program will make ends meet. But he and his team appear to be having a major impact. The Camden Coalition has been able to measure its long-term effect on its first thirty-six super-utilizers. They averaged sixty-two hospital and E.R. visits per month before joining the program and thirty-seven visits after—a forty-per-cent reduction. Their hospital bills averaged $1.2 million per month before and just over half a million after—a fifty-six-per-cent reduction.

These results don’t take into account Brenner’s personnel costs, or the costs of the medications the patients are now taking as prescribed, or the fact that some of the patients might have improved on their own (or died, reducing their costs permanently). The net savings are undoubtedly lower, but they remain, almost certainly, revolutionary. Brenner and his team are out there on the boulevards of Camden demonstrating the possibilities of a strange new approach to health care: to look for the most expensive patients in the system and then direct resources and brainpower toward helping them.

Jeff Brenner has not been the only one to recognize the possibilities in focussing on the hot spots of medicine. One Friday afternoon, I drove to an industrial park on the outskirts of Boston, where a rapidly growing data-analysis company called Verisk Health occupies a floor of a nondescript office complex. It supplies “medical intelligence” to organizations that pay for health benefits—self-insured businesses, many public employers, even the government of Abu Dhabi.

Privacy laws prevent U.S. employers from looking at the details of their employees’ medical spending. So they hand their health-care payment data over to companies that analyze the patterns and tell them how to reduce their health-insurance spending. Mostly, these companies give financial advice on changing benefits—telling them, say, to increase employee co-payments for brand-name drugs or emergency-room visits. But even employers who cut benefits find that their costs continue to outpace their earnings. Verisk, whose clients pay health-care bills for fifteen million patients, is among the data companies that are trying a more sophisticated approach.

Besides the usual statisticians and economists, Verisk recruited doctors to dive into the data. I met one of them, Nathan Gunn, who was thirty-six years old, had completed his medical training at the University of California, San Francisco, and was practicing as an internist part time. The rest of his time he worked as Verisk’s head of research. Mostly, he was in meetings or at his desk poring through “data runs” from clients. He insisted that it was every bit as absorbing as seeing sick patients—sometimes more so. Every data run tells a different human story, he said.

At his computer, he pulled up a data set for me, scrubbed of identifying information, from a client that manages health-care benefits for some two hundred and fifty employers—school districts, a large church association, a bus company, and the like. They had a hundred thousand “covered lives” in all. Payouts for those people rose eight per cent a year, at least three times as fast as the employers’ earnings. This wasn’t good, but the numbers seemed pretty dry and abstract so far. Then he narrowed the list to the top five per cent of spenders—just five thousand people accounted for almost sixty per cent of the spending—and he began parsing further.

“Take two ten-year-old boys with asthma,” he said. “From a disease standpoint, they’re exactly the same cost, right? Wrong. Imagine one of those kids never fills his inhalers and has been in urgent care with asthma attacks three times over the last year, probably because Mom and Dad aren’t really on top of it.” That’s the sort of patient Gunn uses his company’s medical-intelligence software program to zero in on—a patient who is sick and getting inadequate care. “That’s really the sweet spot for preventive care,” Gunn said.

He pulled up patients with known coronary-artery disease. There were nine hundred and twenty-one, he said, reading off the screen. He clicked a few more times and raised his eyebrows. One in seven of them had not had a full office visit with a physician in more than a year. “You can do something about that,” he said.

“Let’s do the E.R.-visit game,” he went on. “This is a fun one.” He sorted the patients by number of visits, much as Jeff Brenner had done for Camden. In this employed population, the No. 1 patient was a twenty-five-year-old woman. In the past ten months, she’d had twenty-nine E.R. visits, fifty-one doctor’s office visits, and a hospital admission.

“I can actually drill into these claims,” he said, squinting at the screen. “All these claims here are migraine, migraine, migraine, migraine, headache, headache, headache.” For a twenty-five-year-old with her profile, he said, medical payments for the previous ten months would be expected to total twenty-eight hundred dollars. Her actual payments came to more than fifty-two thousand dollars—for “headaches.”

Was she a drug seeker? He pulled up her prescription profile, looking for narcotic prescriptions. Instead, he found prescriptions for insulin (she was apparently diabetic) and imipramine, an anti-migraine treatment. Gunn was struck by how faithfully she filled her prescriptions. She hadn’t missed a single renewal—“which is actually interesting,” he said. That’s not what you usually find at the extreme of the cost curve.

The story now became clear to him. She suffered from terrible migraines. She took her medicine, but it wasn’t working. When the headaches got bad, she’d go to the emergency room or to urgent care. The doctors would do CT and MRI scans, satisfy themselves that she didn’t have a brain tumor or an aneurysm, give her a narcotic injection to stop the headache temporarily, maybe renew her imipramine prescription, and send her home, only to have her return a couple of weeks later and see whoever the next doctor on duty was. She wasn’t getting what she needed for adequate migraine care—a primary physician taking her in hand, trying different medications in a systematic way, and figuring out how to better keep her headaches at bay.

As he sorts through such stories, Gunn usually finds larger patterns, too. He told me about an analysis he had recently done for a big information-technology company on the East Coast. It provided health benefits to seven thousand employees and family members, and had forty million dollars in “spend.” The firm had already raised the employees’ insurance co-payments considerably, hoping to give employees a reason to think twice about unnecessary medical visits, tests, and procedures—make them have some “skin in the game,” as they say. Indeed, almost every category of costly medical care went down: doctor visits, emergency-room and hospital visits, drug prescriptions. Yet employee health costs continued to rise—climbing almost ten per cent each year. The company was baffled.

Gunn’s team took a look at the hot spots. The outliers, it turned out, were predominantly early retirees. Most had multiple chronic conditions—in particular, coronary-artery disease, asthma, and complex mental illness. One had badly worsening heart disease and diabetes, and medical bills over two years in excess of eighty thousand dollars. The man, dealing with higher co-payments on a fixed income, had cut back to filling only half his medication prescriptions for his high cholesterol and diabetes. He made few doctor visits. He avoided the E.R.—until a heart attack necessitated emergency surgery and left him disabled with chronic heart failure.

The higher co-payments had backfired, Gunn said. While medical costs for most employees flattened out, those for early retirees jumped seventeen per cent. The sickest patients became much more expensive because they put off care and prevention until it was too late.

The critical flaw in our health-care system that people like Gunn and Brenner are finding is that it was never designed for the kind of patients who incur the highest costs. Medicine’s primary mechanism of service is the doctor visit and the E.R. visit. (Americans make more than a billion such visits each year, according to the Centers for Disease Control.) For a thirty-year-old with a fever, a twenty-minute visit to the doctor’s office may be just the thing. For a pedestrian hit by a minivan, there’s nowhere better than an emergency room. But these institutions are vastly inadequate for people with complex problems: the forty-year-old with drug and alcohol addiction; the eighty-four-year-old with advanced Alzheimer’s disease and a pneumonia; the sixty-year-old with heart failure, obesity, gout, a bad memory for his eleven medications, and half a dozen specialists recommending different tests and procedures. It’s like arriving at a major construction project with nothing but a screwdriver and a crane.

Outsiders tend to be the first to recognize the inadequacies of our social institutions. But, precisely because they are outsiders, they are usually in a poor position to fix them. Gunn, though a doctor, mostly works for people who do not run health systems—employers and insurers. So he counsels them about ways to tinker with the existing system. He tells them how to change co-payments and deductibles so they at least aren’t making their cost problems worse. He identifies doctors and hospitals that seem to be providing particularly ineffective care for high-needs patients, and encourages clients to shift contracts. And he often suggests that clients hire case-management companies—a fast-growing industry with telephone banks of nurses offering high-cost patients advice in the hope of making up for the deficiencies of the system.

The strategy works, sort of. Verisk reports that most of its clients can slow the rate at which their health costs rise, at least to some extent. But few have seen decreases, and it’s not obvious that the improvements can be sustained. Brenner, by contrast, is reinventing medicine from the inside. But he does not run a health-care system, and had to give up his practice to sustain his work. He is an outsider on the inside. So you might wonder whether medical hot-spotting can really succeed on a scale that would help large populations. Yet there are signs that it can.

A recent Medicare demonstration program, given substantial additional resources under the new health-care-reform law, offers medical institutions an extra monthly payment to finance the coördination of care for their most chronically expensive beneficiaries. If total costs fall more than five per cent compared with those of a matched set of control patients, the program allows institutions to keep part of the savings. If costs fail to decline, the institutions have to return the monthly payments.

Several hospitals took the deal when the program was offered, in 2006. One was the Massachusetts General Hospital, in Boston. It asked a general internist named Tim Ferris to design the effort. The hospital had twenty-six hundred chronically high-cost patients, who together accounted for sixty million dollars in annual Medicare spending. They were in nineteen primary-care practices, and Ferris and his team made sure that each had a nurse whose sole job was to improve the coördination of care for these patients. The doctors saw the patients as usual. In between, the nurses saw them for longer visits, made surveillance phone calls, and, in consultation with the doctors, tried to recognize and address problems before they resulted in a hospital visit.

Three years later, hospital stays and trips to the emergency room have dropped more than fifteen per cent. The hospital hit its five-per-cent cost-reduction target. And the team is just getting the hang of what it can do.

Recently, I visited an even more radically redesigned physician practice, in Atlantic City. Cross the bridge into town (Atlantic City is on an island, I learned), ignore the Trump Plaza and Caesars casinos looming ahead of you, drive a few blocks along the Monopoly-board streets (the game took its street names from here), turn onto Tennessee Avenue, and enter the doctors’ office building that’s across the street from the ninety-nine-cent store and the city’s long-shuttered supermarket. On the second floor, just past the occupational-health clinic, you will find the Special Care Center. The reception area, with its rustic taupe upholstery and tasteful lighting, looks like any other doctors’ office. But it houses an experiment started in 2007 by the health-benefit programs of the casino workers’ union and of a hospital, AtlantiCare Medical Center, the city’s two largest pools of employees.

Both are self-insured—they are large enough to pay for their workers’ health care directly—and both have been hammered by the exploding costs. Yes, even hospitals are having a hard time paying their employees’ medical bills. As for the union, its contracts are frequently for workers’ total compensation—wages plus benefits. It gets a fixed pot. Year after year, the low-wage busboys, hotel cleaners, and kitchen staff voted against sacrificing their health benefits. As a result, they have gone without a wage increase for years. Out of desperation, the union’s health fund and the hospital decided to try something new. They got a young Harvard internist named Rushika Fernandopulle to run a clinic exclusively for workers with exceptionally high medical expenses.

Fernandopulle, who was born in Sri Lanka and raised in Baltimore, doesn’t seem like a radical when you meet him. He’s short and round-faced, smiles a lot, and displays two cute rabbit teeth as he tells you how ridiculous the health-care system is and how he plans to change it all. Jeff Brenner was on his advisory board, along with others who have pioneered the concept of intensive outpatient care for complex high-needs patients. The hospital provided the floor space. Fernandopulle created a point system to identify employees likely to have high recurrent costs, and they were offered the chance to join the new clinic.

The Special Care Center reinvented the idea of a primary-care clinic in almost every way. The union’s and the hospital’s health funds agreed to switch from paying the doctors for every individual office visit and treatment to paying a flat monthly fee for each patient. That cut the huge expense that most clinics incur from billing paperwork. The patients were given unlimited access to the clinic without charges—no co-payments, no insurance bills. This, Fernandopulle explained, would force doctors on staff to focus on service, in order to retain their patients and the fees they would bring.

The payment scheme also allowed him to design the clinic around the things that sick, expensive patients most need and value, rather than the ones that pay the best. He adopted an open-access scheduling system to guarantee same-day appointments for the acutely ill. He customized an electronic information system that tracks whether patients are meeting their goals. And he staffed the clinic with people who would help them do it. One nurse practitioner, for instance, was responsible for trying to get every smoker to quit.

I got a glimpse of how unusual the clinic is when I sat in on the staff meeting it holds each morning to review the medical issues of the patients on the appointment books. There was, for starters, the very existence of the meeting. I had never seen this kind of daily huddle at a doctor’s office, with clinicians popping open their laptops and pulling up their patient lists together. Then there was the particular mixture of people who squeezed around the conference table. As in many primary-care offices, the staff had two physicians and two nurse practitioners. But a full-time social worker and the front-desk receptionist joined in for the patient review, too. And, outnumbering them all, there were eight full-time “health coaches.”

Fernandopulle created the position. Each health coach works with patients—in person, by phone, by e-mail—to help them manage their health. Fernandopulle got the idea from the promotoras, community health workers, whom he had seen on a medical mission in the Dominican Republic. The coaches work with the doctors but see their patients far more frequently than the doctors do, at least once every two weeks. Their most important attribute, Fernandopulle explained, is a knack for connecting with sick people, and understanding their difficulties. Most of the coaches come from their patients’ communities and speak their languages. Many have experience with chronic illness in their own families. (One was himself a patient in the clinic.) Few had clinical experience. I asked each of the coaches what he or she had done before working in the Special Care Center. One worked the register at a Dunkin’ Donuts. Another was a Sears retail manager. A third was an administrative assistant at a casino.

“We recruit for attitude and train for skill,” Fernandopulle said. “We don’t recruit from health care. This kind of care requires a very different mind-set from usual care. For example, what is the answer for a patient who walks up to the front desk with a question? The answer is ‘Yes.’ ‘Can I see a doctor?’ ‘Yes.’ ‘Can I get help making my ultrasound appointment?’ ‘Yes.’ Health care trains people to say no to patients.” He told me that he’d had to replace half of the clinic’s initial hires—including a doctor—because they didn’t grasp the focus on patient service.

In forty-five minutes, the staff did a rapid run-through of everyone’s patients. They reviewed the requests that patients had made by e-mail or telephone, the plans for the ones who had appointments that day. Staff members made sure that all patients who made a sick visit the day before got a follow-up call within twenty-four hours, that every test ordered was reviewed, that every unexpected problem was addressed.

Most patients required no more than a ten-second mention. Mr. Green didn’t turn up for his cardiac testing or return calls about it. “I know where his wife works. I’ll track her down,” the receptionist said. Ms. Blue is pregnant and on a high-blood-pressure medication that’s unsafe in pregnancy. “I’ll change her prescription right now,” her doctor said, and keyed it in. A handful of patients required longer discussion. One forty-five-year-old heart-disease patient had just had blood tests that showed worsening kidney failure. The team decided to repeat the blood tests that morning, organize a kidney ultrasound in the afternoon if the tests confirmed the finding, and have him seen in the office at the end of the day.

A staff member read out the hospital census. Of the clinic’s twelve hundred chronically ill patients, just one was in the hospital, and she was being discharged. The clinic’s patients had gone four days without a single E.R. visit. On hearing this news, staffers cheered and broke into applause.

Afterward, I met a patient, Vibha Gandhi. She was fifty-seven years old and had joined the clinic after suffering a third heart attack. She and her husband, Bharat, are Indian immigrants. He cleans casino bathrooms for thirteen dollars an hour on the night shift. Vibha has long had poor health, with diabetes, obesity, and congestive heart failure, but things got much worse in the summer of 2009. A heart attack landed her in intensive care, and her coronary-artery disease proved so advanced as to be inoperable. She arrived in a wheelchair for her first clinic visit. She could not walk more than a few steps without losing her breath and getting a viselike chest pain. The next step for such patients is often a heart transplant.

A year and a half later, she is out of her wheelchair. She attends the clinic’s Tuesday yoga classes. With the help of a walker, she can go a quarter mile without stopping. Although her condition is still fragile—she takes a purseful of medications, and a bout of the flu would send her back to an intensive-care unit—her daily life is far better than she once imagined.

“I didn’t think I would live this long,” Vibha said through Bharat, who translated her Gujarati for me. “I didn’t want to live.”

I asked her what had made her better. The couple credited exercise, dietary changes, medication adjustments, and strict monitoring of her diabetes.

But surely she had been encouraged to do these things after her first two heart attacks. What made the difference this time?

“Jayshree pushes her, and she listens to her only and not to me,” Bharat said.

“Why do you listen to Jayshree?” I asked Vibha.

“Because she talks like my mother,” she said.

Fernandopulle carefully tracks the statistics of those twelve hundred patients. After twelve months in the program, he found, their emergency-room visits and hospital admissions were reduced by more than forty per cent. Surgical procedures were down by a quarter. The patients were also markedly healthier. Among five hundred and three patients with high blood pressure, only two were in poor control. Patients with high cholesterol had, on average, a fifty-point drop in their levels. A stunning sixty-three per cent of smokers with heart and lung disease quit smoking. In surveys, service and quality ratings were high.

But was the program saving money? The team, after all, was more expensive than typical primary care. And certain costs shot up. Because patients took their medications more consistently, drug costs were higher. The doctors ordered more mammograms and diagnostic tests, and caught and treated more cancers and other conditions. There’s also the statistical phenomenon known as “regression to the mean”: the super-high-cost patients may have been on their way to getting better (and less costly) on their own.

So the union’s health fund enlisted an independent economist to evaluate the clinic’s one-year results. According to the data, these workers made up a third of the local union’s costliest ten per cent of members. To determine if the clinic was really making a difference, the economist compared their costs over twelve months with those of a similar group of Las Vegas casino workers. The results, he cautioned, are still preliminary. The sample was small. One patient requiring a heart transplant could wipe away any savings overnight. Nonetheless, compared with the Las Vegas workers, the Atlantic City workers in Fernandopulle’s program experienced a twenty-five-per-cent drop in costs.

And this was just the start. The program, Fernandopulle told me, is still discovering new tricks. His team just recently figured out, for instance, that one reason some patients call 911 for problems the clinic would handle better is that they don’t have the clinic’s twenty-four-hour call number at hand when they need it. The health coaches told the patients to program it into their cell-phone speed dial, but many didn’t know how to do that. So the health coaches began doing it for them, and the number of 911 calls fell. High-cost habits are sticky; staff members are still learning the subtleties of unsticking them.

Their most difficult obstacle, however, has been the waywardness not of patients but of doctors—the doctors whom the patients see outside the clinic. Jeff Brenner’s Camden patients are usually uninsured or on welfare; their doctors were happy to have someone else deal with them. The Atlantic City casino workers and hospital staff, on the other hand, had the best-paying insurance in town. Some doctors weren’t about to let that business slip away.

Fernandopulle told me about a woman who had seen a cardiologist for chest pain two decades ago, when she was in her twenties. It was the result of a temporary, inflammatory condition, but he continued to have her see him for an examination and an electrocardiogram every three months, and a cardiac ultrasound every year. The results were always normal. After the clinic doctors advised her to stop, the cardiologist called her at home to say that her health was at risk if she didn’t keep seeing him. She went back.

The clinic encountered similar troubles with some of the doctors who saw its hospitalized patients. One group of hospital-based internists was excellent, and coördinated its care plans with the clinic. But the others refused, resulting in longer stays and higher costs (and a fee for every visit, while the better group happened to be the only salaried one). When Fernandopulle arranged to direct the patients to the preferred doctors, the others retaliated, trolling the emergency department and persuading the patients to choose them instead.

“ ‘Rogues,’ we call them,” Fernandopulle said. He and his colleagues tried warning the patients about the rogue doctors and contacting the E.R. staff to make sure they knew which doctors were preferred. “One time, we literally pinned a note to a patient, like he was Paddington Bear,” he said. They’ve ended up going to the hospital, and changing the doctors themselves when they have to. As the saying goes, one man’s cost is another man’s income.

The AtlantiCare hospital system is in a curious position in all this. Can it really make sense for a hospital to invest in a program, like the Special Care Center, that aims at reducing hospitalizations, even if its employees are included? I asked David Tilton, the president and C.E.O. of the system, why he was doing it. He had several answers. Some were of the it’s-the-right-thing-to-do variety. But I was interested in the hard-nosed reasons. The Atlantic City economy, he said, could not sustain his health system’s perpetually rising costs. His hospital either fought the pressure to control costs and went down with the local economy or learned how to benefit from cost control.

And there are ways to benefit. At a minimum, a successful hospital could attract patients from competitors, cushioning it against a future in which people need hospitals less. Two decades ago, for instance, Denmark had more than a hundred and fifty hospitals for its five million people. The country then made changes to strengthen the quality and availability of outpatient primary-care services (including payments to encourage physicians to provide e-mail access, off-hours consultation, and nurse managers for complex care). Today, the number of hospitals has shrunk to seventy-one. Within five years, fewer than forty are expected to be required. A smart hospital might position itself to be one of the last ones standing.

Could anything that dramatic happen here? An important idea is getting its test run in America: the creation of intensive outpatient care to target hot spots, and thereby reduce over-all health-care costs. But, if it works, hospitals will lose revenue and some will have to close. Medical companies and specialists profiting from the excess of scans and procedures will get squeezed. This will provoke retaliation, counter-campaigns, intense lobbying for Washington to obstruct reform.

The stats-and-stethoscope upstarts are nonetheless making their dash. Rushika Fernandopulle has set up a version of his Special Care program in Seattle, for Boeing workers, and is developing one in Las Vegas, for casino workers. Nathan Gunn and Verisk Health have landed new contracts during the past year with companies providing health benefits to more than four million employees and family members. Tim Ferris has obtained federal approval to spread his program for Medicare patients to two other hospitals in the Partners Healthcare System, in Boston (including my own). Jeff Brenner, meanwhile, is seeking to lower health-care costs for all of Camden, by getting its primary-care physicians to extend the hot-spot strategy citywide. We’ve been looking to Washington to find out how health-care reform will happen. But people like these are its real leaders.

During my visit to Camden, I attended a meeting that Brenner and several community groups had organized with residents of Northgate II, the building with the highest hospital billing in the city. He wanted to run an idea by them. The meeting took place in the building’s ground-floor lounge. There was juice in Styrofoam cups and potato chips on little red plastic plates. A pastor with the Camden Bible Tabernacle started things off with a prayer. Brenner let one of the other coalition members do the talking.

How much money, he asked, did the residents think had been spent on emergency-room and hospital visits in the past five years for the people in this one building? They had no idea. He wrote out the numbers on an easel pad, but they were imponderable abstractions. The residents’ eyes widened only when he said that the payments, even accounting for unpaid bills, added up to almost sixty thousand dollars per person. He asked how many of them believed that they had received sixty thousand dollars’ worth of health care. That was when the stories came out: the doctors who wouldn’t give anyone on Medicaid an office appointment; the ten-hour emergency-room waits for ten minutes with an intern.

Brenner was proposing to open a doctor’s office right in their building, which would reduce their need for hospital visits. If it delivered better care and saved money, the doctor’s office would receive part of the money that it saved Medicare and Medicaid, and would be able to add services—services that the residents could help choose. With enough savings, they could have same-day doctor visits, nurse practitioners at night, a social worker, a psychologist. When Brenner’s scenario was described, residents murmured approval, but the mention of a social worker brought questions.

“Is she going to be all up in my business?” a woman asked. “I don’t know if I like that. I’m not sure I want a social worker hanging around here.”

This doctor’s office, people were slowly realizing, would be involved in their lives—a medical professional would be after them about their smoking, drinking, diet, medications. That was O.K. if the person were Dr. Brenner. They knew him. They believed that he cared about them. Acceptance, however, would clearly depend upon execution; it wasn’t guaranteed. There was similar ambivalence in the neighborhoods that Compstat strategists targeted for additional—and potentially intrusive—policing.

Yet the stakes in health-care hot-spotting are enormous, and go far beyond health care. A recent report on more than a decade of education-reform spending in Massachusetts detailed a story found in every state. Massachusetts sent nearly a billion dollars to school districts to finance smaller class sizes and better teachers’ pay, yet every dollar ended up being diverted to covering rising health-care costs. For each dollar added to school budgets, the costs of maintaining teacher health benefits took a dollar and forty cents.

Every country in the world is battling the rising cost of health care. No community anywhere has demonstrably lowered its health-care costs (not just slowed their rate of increase) by improving medical services. They’ve lowered costs only by cutting or rationing them. To many people, the problem of health-care costs is best encapsulated in a basic third-grade lesson: you can’t have it all. You want higher wages, lower taxes, less debt? Then cut health-care services.

People like Jeff Brenner are saying that we can have it all—teachers and health care. To be sure, uncertainties remain. Their small, localized successes have not yet been replicated in large populations. Up to a fourth of their patients face problems of a kind they have avoided tackling so far: catastrophic conditions. These are the patients who are in the top one per cent of costs because they were in a car crash that resulted in a hundred thousand dollars in surgery and intensive-care expenses, or had a cancer requiring seven thousand dollars a week for chemo and radiation. There’s nothing much to be done for those patients, you’d think. Yet they are also victims of poor and disjointed service. Improving the value of the services—rewarding better results per dollar spent—could lead to dramatic innovations in catastrophic care, too.

The new health-reform law—Obamacare—is betting big on the Brenners of the world. It says that we can afford to subsidize insurance for millions, remove the ability of private and public insurers to cut high-cost patients from their rolls, and improve the quality of care. The law authorizes new forms of Medicare and Medicaid payment to encourage the development of “medical homes” and “accountable care organizations”—doctors’ offices and medical systems that get financial benefits for being more accessible to patients, better organized, and accountable for reducing the over-all costs of care. Backers believe that, given this support, innovators like Brenner will transform health care everywhere.

Critics say that it’s a pipe dream—more money down the health-care sinkhole. They could turn out to be right, Brenner told me; a well-organized opposition could scuttle efforts like his. “In the next few years, we’re going to have absolutely irrefutable evidence that there are ways to reduce health-care costs, and they are ‘high touch’ and they are at the level of care,” he said. “We are going to know that, hands down, this is possible.” From that point onward, he said, “it’s a political problem.” The struggle will be to survive the obstruction of lobbies, and the partisan tendency to view success as victory for the other side.

Already, these forces of resistance have become Brenner’s prime concern. He needs state legislative approval to bring his program to Medicaid patients at Northgate II and across Camden. He needs federal approval to qualify as an accountable care organization for the city’s Medicare patients. In Camden, he has built support across a range of groups, from the state Chamber of Commerce to local hospitals to activist organizations. But for months—even as rising health costs and shrinking state aid have forced the city to contemplate further school cuts and the layoff of almost half of its police—he has been stalled. With divided branches at both the state and the federal level, “government just gets paralyzed,” he says.

In the meantime, though, he’s forging ahead. In December, he introduced an expanded computer database that lets Camden doctors view laboratory results, radiology reports, emergency-room visits, and discharge summaries for their patients from all the hospitals in town—and could show cost patterns, too. The absence of this sort of information is a daily impediment to the care of patients in Boston, where I practice. Right now, we’re nowhere close to having such data. But this, I’m sure, will change. For in places like Camden, New Jersey, one of the poorest cities in America, there are people showing the way. ♦

Wednesday, January 26, 2011

This past June, Alan Rusbridger, the editor of The Guardian, phoned me and asked, mysteriously, whether I had any idea how to arrange a secure communication. Not really, I confessed. The Times doesn’t have encrypted phone lines, or a Cone of Silence. Well then, he said, he would try to speak circumspectly. In a roundabout way, he laid out an unusual proposition: an organization called WikiLeaks, a secretive cadre of antisecrecy vigilantes, had come into possession of a substantial amount of classified United States government communications. WikiLeaks’s leader, Julian Assange, an eccentric former computer hacker of Australian birth and no fixed residence, offered The Guardian half a million military dispatches from the battlefields of Afghanistan and Iraq. There might be more after that, including an immense bundle of confidential diplomatic cables. The Guardian suggested — to increase the impact as well as to share the labor of handling such a trove — that The New York Times be invited to share this exclusive bounty. The source agreed. Was I interested?

I was interested.

The adventure that ensued over the next six months combined the cloak-and-dagger intrigue of handling a vast secret archive with the more mundane feat of sorting, searching and understanding a mountain of data. As if that were not complicated enough, the project also entailed a source who was elusive, manipulative and volatile (and ultimately openly hostile to The Times and The Guardian); an international cast of journalists; company lawyers committed to keeping us within the bounds of the law; and an array of government officials who sometimes seemed as if they couldn’t decide whether they wanted to engage us or arrest us. By the end of the year, the story of this wholesale security breach had outgrown the story of the actual contents of the secret documents and generated much breathless speculation that something — journalism, diplomacy, life as we know it — had profoundly changed forever.

Soon after Rusbridger’s call, we sent Eric Schmitt, from our Washington bureau, to London. Schmitt has covered military affairs expertly for years, has read his share of classified military dispatches and has excellent judgment and an unflappable demeanor. His main assignment was to get a sense of the material. Was it genuine? Was it of public interest? He would also report back on the proposed mechanics of our collaboration with The Guardian and the German magazine Der Spiegel, which Assange invited as a third guest to his secret smorgasbord. Schmitt would also meet the WikiLeaks leader, who was known to a few Guardian journalists but not to us.

Schmitt’s first call back to The Times was encouraging. There was no question in his mind that the Afghanistan dispatches were genuine. They were fascinating — a diary of a troubled war from the ground up. And there were intimations of more to come, especially classified cables from the entire constellation of American diplomatic outposts. WikiLeaks was holding those back for now, presumably to see how this venture with the establishment media worked out. Over the next few days, Schmitt huddled in a discreet office at The Guardian, sampling the trove of war dispatches and discussing the complexities of this project: how to organize and study such a voluminous cache of information; how to securely transport, store and share it; how journalists from three very different publications would work together without compromising their independence; and how we would all assure an appropriate distance from Julian Assange. We regarded Assange throughout as a source, not as a partner or collaborator, but he was a man who clearly had his own agenda.

By the time of the meetings in London, WikiLeaks had already acquired a measure of international fame or, depending on your point of view, notoriety. Shortly before I got the call from The Guardian, The New Yorker published a rich and colorful profile of Assange, by Raffi Khatchadourian, who had embedded with the group. WikiLeaks’s biggest coup to that point was the release, last April, of video footage taken from one of two U.S. helicopters involved in firing down on a crowd and a building in Baghdad in 2007, killing at least 18 people. While some of the people in the video were armed, others gave no indication of menace; two were in fact journalists for the news agency Reuters. The video, with its soundtrack of callous banter, was horrifying to watch and was an embarrassment to the U.S. military. But in its zeal to make the video a work of antiwar propaganda, WikiLeaks also released a version that didn’t call attention to an Iraqi who was toting a rocket-propelled grenade and packaged the manipulated version under the tendentious rubric “Collateral Murder.” (See the edited and non-edited videos here.)

Throughout our dealings, Assange was coy about where he obtained his secret cache. But the suspected source of the video, as well as the military dispatches and the diplomatic cables to come, was a disillusioned U.S. Army private first class named Bradley Manning, who had been arrested and was being kept in solitary confinement.

On the fourth day of the London meeting, Assange slouched into The Guardian office, a day late. Schmitt took his first measure of the man who would be a large presence in our lives. “He’s tall — probably 6-foot-2 or 6-3 — and lanky, with pale skin, gray eyes and a shock of white hair that seizes your attention,” Schmitt wrote to me later. “He was alert but disheveled, like a bag lady walking in off the street, wearing a dingy, light-colored sport coat and cargo pants, dirty white shirt, beat-up sneakers and filthy white socks that collapsed around his ankles. He smelled as if he hadn’t bathed in days.”

Assange shrugged a huge backpack off his shoulders and pulled out a stockpile of laptops, cords, cellphones, thumb drives and memory sticks that held the WikiLeaks secrets.

The reporters had begun preliminary work on the Afghanistan field reports, using a large Excel spreadsheet to organize the material, then plugging in search terms and combing the documents for newsworthy content. They had run into a puzzling incongruity: Assange said the data included dispatches from the beginning of 2004 through the end of 2009, but the material on the spreadsheet ended abruptly in April 2009. A considerable amount of material was missing. Assange, slipping naturally into the role of office geek, explained that they had hit the limits of Excel. Open a second spreadsheet, he instructed. They did, and the rest of the data materialized — a total of 92,000 reports from the battlefields of Afghanistan.

The reporters came to think of Assange as smart and well educated, extremely adept technologically but arrogant, thin-skinned, conspiratorial and oddly credulous. At lunch one day in The Guardian’s cafeteria, Assange recounted with an air of great conviction a story about the archive in Germany that contains the files of the former Communist secret police, the Stasi. This office, Assange asserted, was thoroughly infiltrated by former Stasi agents who were quietly destroying the documents they were entrusted with protecting. The Der Spiegel reporter in the group, John Goetz, who has reported extensively on the Stasi, listened in amazement. That’s utter nonsense, he said. Some former Stasi personnel were hired as security guards in the office, but the records were well protected.

Assange was openly contemptuous of the American government and certain that he was a hunted man. He told the reporters that he had prepared a kind of doomsday option. He had, he said, distributed highly encrypted copies of his entire secret archive to a multitude of supporters, and if WikiLeaks was shut down, or if he was arrested, he would disseminate the key to make the information public.

Schmitt told me that for all Assange’s bombast and dark conspiracy theories, he had a bit of Peter Pan in him. One night, when they were all walking down the street after dinner, Assange suddenly started skipping ahead of the group. Schmitt and Goetz stared, speechless. Then, just as suddenly, Assange stopped, got back in step with them and returned to the conversation he had interrupted.

For the rest of the week Schmitt worked with David Leigh, The Guardian’s investigations editor; Nick Davies, an investigative reporter for the paper; and Goetz, of Der Spiegel, to organize and sort the material. With help from two of The Times’s best computer minds — Andrew Lehren and Aron Pilhofer — they figured out how to assemble the material into a conveniently searchable and secure database.

Journalists are characteristically competitive, but the group worked well together. They brainstormed topics to explore and exchanged search results. Der Spiegel offered to check the logs against incident reports submitted by the German Army to its Parliament — partly as story research, partly as an additional check on authenticity.

Assange provided us the data on the condition that we not write about it before specific dates that WikiLeaks planned on posting the documents on a publicly accessible Web site. The Afghanistan documents would go first, after we had a few weeks to search the material and write our articles. The larger cache of Iraq-related documents would go later. Such embargoes — agreements not to publish information before a set date — are commonplace in journalism. Everything from studies in medical journals to the annual United States budget is released with embargoes. They are a constraint with benefits, the principal one being the chance to actually read and reflect on the material before publishing it into public view. As Assange surely knew, embargoes also tend to build suspense and amplify a story, especially when multiple news outlets broadcast it at once. The embargo was the only condition WikiLeaks would try to impose on us; what we wrote about the material was entirely up to us. Much later, some American news outlets reported that they were offered last-minute access to WikiLeaks documents if they signed contracts with financial penalties for early disclosure. The Times was never asked to sign anything or to pay anything. For WikiLeaks, at least in this first big venture, exposure was its own reward.

Back in New York we assembled a team of reporters, data experts and editors and quartered them in an out-of-the-way office. Andrew Lehren, of our computer-assisted-reporting unit, did the first cut, searching terms on his own or those suggested by other reporters, compiling batches of relevant documents and summarizing the contents. We assigned reporters to specific areas in which they had expertise and gave them password access to rummage in the data. This became the routine we would follow with subsequent archives.

An air of intrigue verging on paranoia permeated the project, perhaps understandably, given that we were dealing with a mass of classified material and a source who acted like a fugitive, changing crash pads, e-mail addresses and cellphones frequently. We used encrypted Web sites. Reporters exchanged notes via Skype, believing it to be somewhat less vulnerable to eavesdropping. On conference calls, we spoke in amateurish code. Assange was always “the source.” The latest data drop was “the package.” When I left New York for two weeks to visit bureaus in Pakistan and Afghanistan, where we assume that communications may be monitored, I was not to be copied on message traffic about the project. I never imagined that any of this would defeat a curious snoop from the National Security Agency or Pakistani intelligence. And I was never entirely sure whether that prospect made me more nervous than the cyberwiles of WikiLeaks itself. At a point when relations between the news organizations and WikiLeaks were rocky, at least three people associated with this project had inexplicable activity in their e-mail that suggested someone was hacking into their accounts.

From consultations with our lawyers, we were confident that reporting on the secret documents could be done within the law, but we speculated about what the government — or some other government — might do to impede our work or exact recriminations. And, the law aside, we felt an enormous moral and ethical obligation to use the material responsibly. While we assumed we had little or no ability to influence what WikiLeaks did, let alone what would happen once this material was loosed in the echo chamber of the blogosphere, that did not free us from the need to exercise care in our own journalism. From the beginning, we agreed that in our articles and in any documents we published from the secret archive, we would excise material that could put lives at risk.

Guided by reporters with extensive experience in the field, we redacted the names of ordinary citizens, local officials, activists, academics and others who had spoken to American soldiers or diplomats. We edited out any details that might reveal ongoing intelligence-gathering operations, military tactics or locations of material that could be used to fashion terrorist weapons. Three reporters with considerable experience of handling military secrets — Eric Schmitt, Michael Gordon and C. J. Chivers — went over the documents we considered posting. Chivers, an ex-Marine who has reported for us from several battlefields, brought a practiced eye and cautious judgment to the business of redaction. If a dispatch noted that Aircraft A left Location B at a certain time and arrived at Location C at a certain time, Chivers edited it out on the off chance that this could teach enemy forces something useful about the capabilities of that aircraft.

The ﬁrst articles in the project, which we called the War Logs, were scheduled to go up on the Web sites of The Times, The Guardian and Der Spiegel on Sunday, July 25. We approached the White House days before that to get its reaction to the huge breach of secrecy as well as to specific articles we planned to write — including a major one about Pakistan’s ambiguous role as an American ally. On July 24, the day before the War Logs went live, I attended a farewell party for Roger Cohen, a columnist for The Times and The International Herald Tribune, that was given by Richard Holbrooke, the Obama administration’s special envoy to Afghanistan and Pakistan. A voracious consumer of inside information, Holbrooke had a decent idea of what was coming, and he pulled me away from the crowd to show me the fusillade of cabinet-level e-mail ricocheting through his BlackBerry, thus demonstrating both the frantic anxiety in the administration and, not incidentally, the fact that he was very much in the loop. The Pakistan article, in particular, would complicate his life. But one of Holbrooke’s many gifts was his ability to make pretty good lemonade out of the bitterest lemons; he was already spinning the reports of Pakistani duplicity as leverage he could use to pull the Pakistanis back into closer alignment with American interests. Five months later, when Holbrooke — just 69, and seemingly indestructible — died of a torn aorta, I remembered that evening. And what I remembered best was that he was as excited to be on the cusp of a big story as I was.

We posted the articles on NYTimes.com the next day at 5 p.m. — a time picked to reconcile the different publishing schedules of the three publications. I was proud of what a crew of great journalists had done to fashion coherent and instructive reporting from a jumble of raw field reports, mostly composed in a clunky patois of military jargon and acronyms. The reporters supplied context, nuance and skepticism. There was much in that first round of articles worth reading, but my favorite single piece was one of the simplest. Chivers gathered all of the dispatches related to a single, remote, beleaguered American military outpost and stitched them together into a heartbreaking narrative. The dispatches from this outpost represent in miniature the audacious ambitions, gradual disillusionment and ultimate disappointment that Afghanistan has dealt to occupiers over the centuries.

If anyone doubted that the three publications operated independently, the articles we posted that day made it clear that we followed our separate muses. The Guardian, which is an openly left-leaning newspaper, used the first War Logs to emphasize civilian casualties in Afghanistan, claiming the documents disclosed that coalition forces killed “hundreds of civilians in unreported incidents,” underscoring the cost of what the paper called a “failing war.” Our reporters studied the same material but determined that all the major episodes of civilian deaths we found in the War Logs had been reported in The Times, many of them on the front page. (In fact, two of our journalists, Stephen Farrell and Sultan Munadi, were kidnapped by the Taliban while investigating one major episode near Kunduz. Munadi was killed during an ensuing rescue by British paratroopers.) The civilian deaths that had not been previously reported came in ones and twos and did not add up to anywhere near “hundreds.” Moreover, since several were either duplicated or missing from the reports, we concluded that an overall tally would be little better than a guess.

Another example: The Times gave prominence to the dispatches reflecting American suspicions that Pakistani intelligence was playing a double game in Afghanistan — nodding to American interests while abetting the Taliban. We buttressed the interesting anecdotal material of Pakistani double-dealing with additional reporting. The Guardian was unimpressed by those dispatches and treated them more dismissively.

Three months later, with the French daily Le Monde added to the group, we published Round 2, the Iraq War Logs, including articles on how the United States turned a blind eye to the torture of prisoners by Iraqi forces working with the U.S., how Iraq spawned an extraordinary American military reliance on private contractors and how extensively Iran had meddled in the conflict.

By this time, The Times’s relationship with our source had gone from wary to hostile. I talked to Assange by phone a few times and heard out his complaints. He was angry that we declined to link our online coverage of the War Logs to the WikiLeaks Web site, a decision we made because we feared — rightly, as it turned out — that its trove would contain the names of low-level informants and make them Taliban targets. “Where’s the respect?” he demanded. “Where’s the respect?” Another time he called to tell me how much he disliked our profile of Bradley Manning, the Army private suspected of being the source of WikiLeaks’s most startling revelations. The article traced Manning’s childhood as an outsider and his distress as a gay man in the military. Assange complained that we “psychologicalized” Manning and gave short shrift to his “political awakening.”

The final straw was a front-page profile of Assange by John Burns and Ravi Somaiya, published Oct. 24, that revealed fractures within WikiLeaks, attributed by Assange’s critics to his imperious management style. Assange denounced the article to me, and in various public forums, as “a smear.”

Assange was transformed by his outlaw celebrity. The derelict with the backpack and the sagging socks now wore his hair dyed and styled, and he favored fashionably skinny suits and ties. He became a kind of cult figure for the European young and leftish and was evidently a magnet for women. Two Swedish women filed police complaints claiming that Assange insisted on having sex without a condom; Sweden’s strict laws on nonconsensual sex categorize such behavior as rape, and a prosecutor issued a warrant to question Assange, who initially described it as a plot concocted to silence or discredit WikiLeaks.

I came to think of Julian Assange as a character from a Stieg Larsson thriller — a man who could figure either as hero or villain in one of the megaselling Swedish novels that mix hacker counterculture, high-level conspiracy and sex as both recreation and violation.

In October, WikiLeaks gave The Guardian its third archive, a quarter of a million communications between the U.S. State Department and its outposts around the globe. This time, Assange imposed a new condition: The Guardian was not to share the material with The New York Times. Indeed, he told Guardian journalists that he opened discussions with two other American news organizations — The Washington Post and the McClatchy chain — and intended to invite them in as replacements for The Times. He also enlarged his recipient list to include El País, the leading Spanish-language newspaper.

The Guardian was uncomfortable with Assange’s condition. By now the journalists from The Times and The Guardian had a good working relationship. The Times provided a large American audience for the revelations, as well as access to the U.S. government for comment and context. And given the potential legal issues and public reaction, it was good to have company in the trenches. Besides, we had come to believe that Assange was losing control of his stockpile of secrets. An independent journalist, Heather Brooke, had obtained material from a WikiLeaks dissident and joined in a loose alliance with The Guardian. Over the coming weeks, batches of cables would pop up in newspapers in Lebanon, Australia and Norway. David Leigh, The Guardian’s investigations editor, concluded that these rogue leaks released The Guardian from any pledge, and he gave us the cables.

On Nov. 1, Assange and two of his lawyers burst into Alan Rusbridger’s office, furious that The Guardian was asserting greater independence and suspicious that The Times might be in possession of the embassy cables. Over the course of an eight-hour meeting, Assange intermittently raged against The Times — especially over our front-page profile — while The Guardian journalists tried to calm him. In midstorm, Rusbridger called me to report on Assange’s grievances and relay his demand for a front-page apology in The Times. Rusbridger knew that this was a nonstarter, but he was buying time for the tantrum to subside. In the end, both he and Georg Mascolo, editor in chief of Der Spiegel, made clear that they intended to continue their collaboration with The Times; Assange could take it or leave it. Given that we already had all of the documents, Assange had little choice. Over the next two days, the news organizations agreed on a timetable for publication.

The following week, we sent Ian Fisher, a deputy foreign editor who was a principal coordinator on our processing of the embassy cables, to London to work out final details. The meeting went smoothly, even after Assange arrived. “Freakishly good behavior,” Fisher e-mailed me afterward. “No yelling or crazy mood swings.” But after dinner, as Fisher was leaving, Assange smirked and offered a parting threat: “Tell me, are you in contact with your legal counsel?” Fisher replied that he was. “You had better be,” Assange said.

Fisher left London with an understanding that we would continue to have access to the material. But just in case, we took out a competitive insurance policy. We had Scott Shane, a Washington correspondent, pull together a long, just-in-case article summing up highlights of the cables, which we could quickly post on our Web site. If WikiLeaks sprang another leak, we would be ready.

Because of the range of the material and the very nature of diplomacy, the embassy cables were bound to be more explosive than the War Logs. Dean Baquet, our Washington bureau chief, gave the White House an early warning on Nov. 19. The following Tuesday, two days before Thanksgiving, Baquet and two colleagues were invited to a windowless room at the State Department, where they encountered an unsmiling crowd. Representatives from the White House, the State Department, the Office of the Director of National Intelligence, the C.I.A., the Defense Intelligence Agency, the F.B.I. and the Pentagon gathered around a conference table. Others, who never identified themselves, lined the walls. A solitary note-taker tapped away on a computer.

The meeting was off the record, but it is fair to say the mood was tense. Scott Shane, one reporter who participated in the meeting, described “an undertone of suppressed outrage and frustration.”

Subsequent meetings, which soon gave way to daily conference calls, were more businesslike. Before each discussion, our Washington bureau sent over a batch of specific cables that we intended to use in the coming days. They were circulated to regional specialists, who funneled their reactions to a small group at State, who came to our daily conversations with a list of priorities and arguments to back them up. We relayed the government’s concerns, and our own decisions regarding them, to the other news outlets.

The administration’s concerns generally fell into three categories. First was the importance of protecting individuals who had spoken candidly to American diplomats in oppressive countries. We almost always agreed on those and were grateful to the government for pointing out some we overlooked.

“We were all aware of dire stakes for some of the people named in the cables if we failed to obscure their identities,” Shane wrote to me later, recalling the nature of the meetings. Like many of us, Shane has worked in countries where dissent can mean prison or worse. “That sometimes meant not just removing the name but also references to institutions that might give a clue to an identity and sometimes even the dates of conversations, which might be compared with surveillance tapes of an American Embassy to reveal who was visiting the diplomats that day.”

The second category included sensitive American programs, usually related to intelligence. We agreed to withhold some of this information, like a cable describing an intelligence-sharing program that took years to arrange and might be lost if exposed. In other cases, we went away convinced that publication would cause some embarrassment but no real harm.

The third category consisted of cables that disclosed candid comments by and about foreign officials, including heads of state. The State Department feared publication would strain relations with those countries. We were mostly unconvinced.

The embassy cables were a different kind of treasure from the War Logs. For one thing, they covered the entire globe — virtually every embassy, consulate and interest section that the United States maintains. They contained the makings of many dozens of stories: candid American appraisals of foreign leaders, narratives of complicated negotiations, allegations of corruption and duplicity, countless behind-the-scenes insights. Some of the material was of narrow local interest; some of it had global implications. Some provided authoritative versions of events not previously fully understood. Some consisted of rumor and flimsy speculation.

Unlike most of the military dispatches, the embassy cables were written in clear English, sometimes with wit, color and an ear for dialogue. (“Who knew,” one of our English colleagues marveled, “that American diplomats could write?”)

Even more than the military logs, the diplomatic cables called for context and analysis. It was important to know, for example, that cables sent from an embassy are routinely dispatched over the signature of the ambassador and those from the State Department are signed by the secretary of state, regardless of whether the ambassador or secretary had actually seen the material. It was important to know that much of the communication between Washington and its outposts is given even more restrictive classification — top secret or higher — and was thus missing from this trove. We searched in vain, for example, for military or diplomatic reports on the fate of Pat Tillman, the former football star and Army Ranger who was killed by friendly fire in Afghanistan. We found no reports on how Osama bin Laden eluded American forces in the mountains of Tora Bora. (In fact, we found nothing but second- and thirdhand rumors about bin Laden.) If such cables exist, they were presumably classified top secret or higher.

And it was important to remember that diplomatic cables are versions of events. They can be speculative. They can be ambiguous. They can be wrong.

One of our first articles drawn from the diplomatic cables, for example, reported on a secret intelligence assessment that Iran had obtained a supply of advanced missiles from North Korea, missiles that could reach European capitals. Outside experts long suspected that Iran obtained missile parts but not the entire weapons, so this glimpse of the official view was revealing. The Washington Post fired back with a different take, casting doubt on whether the missile in question had been transferred to Iran or whether it was even a workable weapon. We went back to the cables — and the experts — and concluded in a subsequent article that the evidence presented “a murkier picture.”

The tension between a newspaper’s obligation to inform and the government’s responsibility to protect is hardly new. At least until this year, nothing The Times did on my watch caused nearly so much agitation as two articles we published about tactics employed by the Bush administration after the attacks of Sept. 11, 2001. The first, which was published in 2005 and won a Pulitzer Prize, revealed that the National Security Agency was eavesdropping on domestic phone conversations and e-mail without the legal courtesy of a warrant. The other, published in 2006, described a vast Treasury Department program to screen international banking records.

I have vivid memories of sitting in the Oval Office as President George W. Bush tried to persuade me and the paper’s publisher to withhold the eavesdropping story, saying that if we published it, we should share the blame for the next terrorist attack. We were unconvinced by his argument and published the story, and the reaction from the government — and conservative commentators in particular — was vociferous.

This time around, the Obama administration’s reaction was different. It was, for the most part, sober and professional. The Obama White House, while strongly condemning WikiLeaks for making the documents public, did not seek an injunction to halt publication. There was no Oval Office lecture. On the contrary, in our discussions before publication of our articles, White House officials, while challenging some of the conclusions we drew from the material, thanked us for handling the documents with care. The secretaries of state and defense and the attorney general resisted the opportunity for a crowd-pleasing orgy of press bashing. There has been no serious official talk — unless you count an ambiguous hint by Senator Joseph Lieberman — of pursuing news organizations in the courts. Though the release of these documents was certainly embarrassing, the relevant government agencies actually engaged with us in an attempt to prevent the release of material genuinely damaging to innocent individuals or to the national interest.

The broader public reaction was mixed — more critical in the first days; more sympathetic as readers absorbed the articles and the sky did not fall; and more hostile to WikiLeaks in the U.S. than in Europe, where there is often a certain pleasure in seeing the last superpower taken down a peg.

In the days after we began our respective series based on the embassy cables, Alan Rusbridger and I went online to answer questions from readers. The Guardian, whose readership is more sympathetic to the guerrilla sensibilities of WikiLeaks, was attacked for being too fastidious about redacting the documents: How dare you censor this material? What are you hiding? Post everything now! The mail sent to The Times, at least in the first day or two, came from the opposite field. Many readers were indignant and alarmed: Who needs this? How dare you? What gives you the right?

Much of the concern reflected a genuine conviction that in perilous times the president needs extraordinary powers, unfettered by Congressional oversight, court meddling or the strictures of international law and certainly safe from nosy reporters. That is compounded by a popular sense that the elite media have become too big for their britches and by the fact that our national conversation has become more polarized and strident.

Although it is our aim to be impartial in our presentation of the news, our attitude toward these issues is far from indifferent. The journalists at The Times have a large and personal stake in the country’s security. We live and work in a city that has been tragically marked as a favorite terrorist target, and in the wake of 9/11 our journalists plunged into the ruins to tell the story of what happened here. Moreover, The Times has nine staff correspondents assigned to the two wars still being waged in the wake of that attack, plus a rotating cast of photographers, visiting writers and scores of local stringers and support staff. They work in this high-risk environment because, while there are many places you can go for opinions about the war, there are few places — and fewer by the day — where you can go to find honest, on-the-scene reporting about what is happening. We take extraordinary precautions to keep them safe, but we have had two of our Iraqi journalists murdered for doing their jobs. We have had four journalists held hostage by the Taliban — two of them for seven months. We had one Afghan journalist killed in a rescue attempt. Last October, while I was in Kabul, we got word that a photographer embedded for us with troops near Kandahar stepped on an improvised mine and lost both his legs.

We are invested in the struggle against murderous extremism in another sense. The virulent hatred espoused by terrorists, judging by their literature, is directed not just against our people and our buildings but also at our values and at our faith in the self-government of an informed electorate. If the freedom of the press makes some Americans uneasy, it is anathema to the ideologists of terror.

So we have no doubts about where our sympathies lie in this clash of values. And yet we cannot let those sympathies transform us into propagandists, even for a system we respect.

I’m the first to admit that news organizations, including this one, sometimes get things wrong. We can be overly credulous (as in some of the prewar reporting about Iraq’s supposed weapons of mass destruction) or overly cynical about official claims and motives. We may err on the side of keeping secrets (President Kennedy reportedly wished, after the fact, that The Times had published what it knew about the planned Bay of Pigs invasion, which possibly would have helped avert a bloody debacle) or on the side of exposing them. We make the best judgments we can. When we get things wrong, we try to correct the record. A free press in a democracy can be messy. But the alternative is to give the government a veto over what its citizens are allowed to know. Anyone who has worked in countries where the news diet is controlled by the government can sympathize with Thomas Jefferson’s oft-quoted remark that he would rather have newspapers without government than government without newspapers.

The intentions of our founders have rarely been as well articulated as they were by Justice Hugo Black 40 years ago, concurring with the Supreme Court ruling that stopped the government from suppressing the secret Vietnam War history called the Pentagon Papers: “The government’s power to censor the press was abolished so that the press would remain forever free to censure the government. The press was protected so that it could bare the secrets of government and inform the people.”

There is no neat formula for maintaining this balance. In practice, the tension between our obligation to inform and the government’s obligation to protect plays out in a set of rituals. As one of my predecessors, Max Frankel, then the Washington bureau chief, wrote in a wise affidavit filed during the Pentagon Papers case: “For the vast majority of ‘secrets,’ there has developed between the government and the press (and Congress) a rather simple rule of thumb: The government hides what it can, pleading necessity as long as it can, and the press pries out what it can, pleading a need and a right to know. Each side in this ‘game’ regularly ‘wins’ and ‘loses’ a round or two. Each fights with the weapons at its command. When the government loses a secret or two, it simply adjusts to a new reality.”

In fact, leaks of classified material — sometimes authorized — are part of the way business is conducted in Washington, as one wing of the bureaucracy tries to one-up another or officials try to shift blame or claim credit or advance or confound a particular policy. For further evidence that our government is highly selective in its approach to secrets, look no further than Bob Woodward’s all-but-authorized accounts of the innermost deliberations of our government.

The government surely cheapens secrecy by deploying it so promiscuously. According to the Pentagon, about 500,000 people have clearance to use the database from which the secret cables were pilfered. Weighing in on the WikiLeaks controversy in The Guardian, Max Frankel remarked that secrets shared with such a legion of “cleared” officials, including low-level army clerks, “are not secret.” Governments, he wrote, “must decide that the random rubber-stamping of millions of papers and computer files each year does not a security system make.”

Beyond the basic question of whether the press should publish secrets, criticism of the WikiLeaks documents generally fell into three themes: 1. That the documents were of dubious value, because they told us nothing we didn’t already know. 2. That the disclosures put lives at risk — either directly, by identifying confidential informants, or indirectly, by complicating our ability to build alliances against terror. 3. That by doing business with an organization like WikiLeaks, The Times and other news organizations compromised their impartiality and independence.

I’m a little puzzled by the complaint that most of the embassy traffic we disclosed did not profoundly change our understanding of how the world works. Ninety-nine percent of what we read or hear on the news does not profoundly change our understanding of how the world works. News mostly advances by inches and feet, not in great leaps. The value of these documents — and I believe they have immense value — is not that they expose some deep, unsuspected perfidy in high places or that they upend your whole view of the world. For those who pay close attention to foreign policy, these documents provide texture, nuance and drama. They deepen and correct your understanding of how things unfold; they raise or lower your estimation of world leaders. For those who do not follow these subjects as closely, the stories are an opportunity to learn more. If a project like this makes readers pay attention, think harder, understand more clearly what is being done in their name, then we have performed a public service. And that does not count the impact of these revelations on the people most touched by them. WikiLeaks cables in which American diplomats recount the extravagant corruption of Tunisia’s rulers helped fuel a popular uprising that has overthrown the government.

As for the risks posed by these releases, they are real. WikiLeaks’s first data dump, the publication of the Afghanistan War Logs, included the names of scores of Afghans that The Times and other news organizations had carefully purged from our own coverage. Several news organizations, including ours, reported this dangerous lapse, and months later a Taliban spokesman claimed that Afghan insurgents had been perusing the WikiLeaks site and making a list. I anticipate, with dread, the day we learn that someone identified in those documents has been killed.

WikiLeaks was roundly criticized for its seeming indifference to the safety of those informants, and in its subsequent postings it has largely followed the example of the news organizations and redacted material that could get people jailed or killed. Assange described it as a “harm minimization” policy. In the case of the Iraq war documents, WikiLeaks applied a kind of robo-redaction software that stripped away names (and rendered the documents almost illegible). With the embassy cables, WikiLeaks posted mostly documents that had already been redacted by The Times and its fellow news organizations. And there were instances in which WikiLeaks volunteers suggested measures to enhance the protection of innocents. For example, someone at WikiLeaks noticed that if the redaction of a phrase revealed the exact length of the words, an alert foreign security service might match the number of letters to a name and affiliation and thus identify the source. WikiLeaks advised everyone to substitute a dozen uppercase X’s for each redacted passage, no matter how long or short.

Whether WikiLeaks’s “harm minimization” is adequate, and whether it will continue, is beyond my power to predict or influence. WikiLeaks does not take guidance from The New York Times. In the end, I can answer only for what my own paper has done, and I believe we have behaved responsibly.

The idea that the mere publication of such a wholesale collection of secrets will make other countries less willing to do business with our diplomats seems to me questionable. Even Defense Secretary Robert Gates called this concern “overwrought.” Foreign governments cooperate with us, he pointed out, not because they necessarily love us, not because they trust us to keep their secrets, but because they need us. It may be that for a time diplomats will choose their words more carefully or circulate their views more narrowly, but WikiLeaks has not repealed the laws of self-interest. A few weeks after we began publishing articles about the embassy cables, David Sanger, our chief Washington correspondent, told me: “At least so far, the evidence that foreign leaders are no longer talking to American diplomats is scarce. I’ve heard about nervous jokes at the beginning of meetings, along the lines of ‘When will I be reading about this conversation?’ But the conversations are happening. . . . American diplomacy has hardly screeched to a halt.”

As for our relationship with WikiLeaks, Julian Assange has been heard to boast that he served as a kind of puppet master, recruiting several news organizations, forcing them to work in concert and choreographing their work. This is characteristic braggadocio — or, as my Guardian colleagues would say, bollocks. Throughout this experience we have treated Assange as a source. I will not say “a source, pure and simple,” because as any reporter or editor can attest, sources are rarely pure or simple, and Assange was no exception. But the relationship with sources is straightforward: you don’t necessarily endorse their agenda, echo their rhetoric, take anything they say at face value, applaud their methods or, most important, allow them to shape or censor your journalism. Your obligation, as an independent news organization, is to verify the material, to supply context, to exercise responsible judgment about what to publish and what not to publish and to make sense of it. That is what we did.

But while I do not regard Assange as a partner, and I would hesitate to describe what WikiLeaks does as journalism, it is chilling to contemplate the possible government prosecution of WikiLeaks for making secrets public, let alone the passage of new laws to punish the dissemination of classified information, as some have advocated. Taking legal recourse against a government official who violates his trust by divulging secrets he is sworn to protect is one thing. But criminalizing the publication of such secrets by someone who has no official obligation seems to me to run up against the First Amendment and the best traditions of this country. As one of my colleagues asks: If Assange were an understated professorial type rather than a character from a missing Stieg Larsson novel, and if WikiLeaks were not suffused with such glib antipathy toward the United States, would the reaction to the leaks be quite so ferocious? And would more Americans be speaking up against the threat of reprisals?

Whether the arrival of WikiLeaks has fundamentally changed the way journalism is made, I will leave to others and to history. Frankly, I think the impact of WikiLeaks on the culture has probably been overblown. Long before WikiLeaks was born, the Internet transformed the landscape of journalism, creating a wide-open and global market with easier access to audiences and sources, a quicker metabolism, a new infrastructure for sharing and vetting information and a diminished respect for notions of privacy and secrecy. Assange has claimed credit on several occasions for creating something he calls “scientific journalism,” meaning that readers are given the raw material to judge for themselves whether the journalistic write-ups are trustworthy. But newspapers have been publishing texts of documents almost as long as newspapers have existed — and ever since the Internet eliminated space restrictions, we have done so copiously.

Nor is it clear to me that WikiLeaks represents some kind of cosmic triumph of transparency. If the official allegations are to be believed, most of WikiLeaks’s great revelations came from a single anguished Army private — anguished enough to risk many years in prison. It’s possible that the creation of online information brokers like WikiLeaks and OpenLeaks, a breakaway site announced in December by a former Assange colleague named Daniel Domscheit-Berg, will be a lure for whistle-blowers and malcontents who fear being caught consorting directly with a news organization like mine. But I suspect we have not reached a state of information anarchy. At least not yet.

As 2010 wound down, The Times and its news partners held a conference call to discuss where we go from here. The initial surge of articles drawn from the secret cables was over. More would trickle out but without a fixed schedule. We agreed to continue the redaction process, and we agreed we would all urge WikiLeaks to do the same. But this period of intense collaboration, and of regular contact with our source, was coming to a close.

Just before Christmas, Ian Katz, The Guardian’s deputy editor, went to see Assange, who had been arrested in London on the Swedish warrant, briefly jailed and bailed out by wealthy admirers and was living under house arrest in a country manor in East Anglia while he fought Sweden’s attempt to extradite him. The flow of donations to WikiLeaks, which he claimed hit 100,000 euros a day at its peak, was curtailed when Visa, MasterCard and PayPal refused to be conduits for contributors — prompting a concerted assault on the Web sites of those companies by Assange’s hacker sympathizers. He would soon sign a lucrative book deal to finance his legal struggles.

The Guardian seemed to have joined The Times on Assange’s enemies list, first for sharing the diplomatic cables with us, then for obtaining and reporting on the unredacted record of the Swedish police complaints against Assange. (Live by the leak. . . .) In his fury at this perceived betrayal, Assange granted an interview to The Times of London, in which he vented his displeasure with our little media consortium. If he thought this would ingratiate him with The Guardian rival, he was naïve. The paper happily splashed its exclusive interview, then followed it with an editorial calling Assange a fool and a hypocrite.

At the mansion in East Anglia, Assange seated Katz before a roaring fire in the drawing room and ruminated for four hours about the Swedish case, his financial troubles and his plan for a next phase of releases. He talked vaguely about secrets still in his quiver, including what he regards as a damning cache of e-mail from inside an American bank.

He spun out an elaborate version of a U.S. Justice Department effort to exact punishment for his assault on American secrecy. If he was somehow extradited to the United States, he said, “I would still have a high chance of being killed in the U.S. prison system, Jack Ruby style, given the continual calls for my murder by senior and influential U.S. politicians.”

While Assange mused darkly in his exile, one of his lawyers sent out a mock Christmas card that suggested at least someone on the WikiLeaks team was not lacking a sense of the absurd.

The message:

“Dear kids,

Santa is Mum & Dad.

Love,

WikiLeaks.”

Bill Keller is the executive editor of The New York Times. This essay is adapted from his introduction to “Open Secrets: WikiLeaks, War and American Diplomacy: Complete and Updated Coverage from The New York Times,” which will be published in e-book form on Jan. 31. For sale at nytimes.com/opensecrets.