This article was taken from the September 2012 issue of
Wired magazine. Be the first to read Wired's articles in print
before they're posted online, and get your hands on loads of
additional content bysubscribing
online.

On the afternoon of February
15, 2011, Jaime Zapata, a 32-year-old special agent with US
Immigration and Customs Enforcement, was shot dead by members of a
drug cartel as he drove along a four-lane highway in
Mexico.

Zapata's partner, Victor Ávila, 38,
who survived the attack, later said that as many as 15 gunmen
opened fire with automatic rifles, even though Zapata had
identified himself as a diplomat and the armoured SUV in which the
pair were riding had number plates identifying it as an official
vehicle.

Back in Washington DC, senior US
administration officials wanted revenge for what they saw as the
deliberate killing of a federal agent and the attempted murder of
another. Michele Leonhart, head of the US Drug Enforcement
Administration (DEA) and a veteran of the drug wars, asked, "What
can we do to make an immediate impact against the cartels, to send
a message?" recalls Derek Maltz, the special agent in charge of the
DEA's Special Operations Division. "We decided to do a
law-enforcement strike," Maltz says.

To conduct what became known as
Operation Fallen Hero, investigators turned to a little-known
Silicon Valley software company called Palantir Technologies.
Palantir's expertise is in finding connections between people,
places and events in large repositories of electronic data. Federal
agents had a trove of reporting on the drug cartels, their members,
their funding mechanisms and smuggling routes. They had dossiers,
informants' reports, surveillance images, intercepted electronic
communications, footage from drones. But investigators lacked a way
to assemble and share all that intelligence with one another, and
to quickly find leads buried in mountains of
information.

Investigators with Zapata's agency bought Palantir's
software, plugged it into their databases and used it to track down
members of the cartel. The results were astonishing. Palantir
helped to identify connections among key individuals and
organisations. Officials reported that this kind of painstaking
detective work -- reading reports, piecing together clues, drawing
links between people -- would have taken months without
technological assistance. With the help of Palantir, large amounts
of data from disparate sources were analysed within days.

Law-enforcement officers across the
US, Mexico and South America confiscated 467 kilograms of cocaine,
30 kilograms of methamphetamine and 282 weapons, and arrested 676
people -- including the cartel member suspected of killing
Zapata.

Officials were so impressed with Palantir's software
that seven months later they bought licences for 1,150
investigators and analysts across the country. The price, including
training, was £4.8 million a year. The government chose not to seek
a bid from some of Palantir's competitors because, officials said,
analysts had already tried three products and each "failed to
provide the necessary comprehensive solution on missions where our
agents risk life and limb".

As far as Washington was concerned,
only Palantir would do.

Such an endorsement would be
remarkable if it were unique.But over the
past three years, Palantir, whose office in Tysons Corner,
Virginia, is just ten kilometres from the CIA's headquarters, has
become a darling of the US law-enforcement and national-security
establishment -- and its business in the UK is growing. US national
security agencies now use Palantir for numerous variation on the
challenge that bedevilled analysts in Operation Fallen Hero -- how
to organise and catalogue intimidating amounts of data and then
find meaningful insights that defy humans alone.

In the US, Palantir has sold its
software to the CIA, the military's Special Command and the Marine
Corps. The FBI, the Defense Intelligence Agency, the National
Counterterrorism Center and the Department of Homeland Security are
all customers. The director of the National Security Agency has
said that Palantir's software could help the agency to "see" into
cyberspace to defend against hackers and spies attempting to breach
government computer networks. The Los Angeles Police Department
uses Palantir. So does the New York Police Department, whose
intelligence and counterterrorism unit rivals the sophistication of
the FBI and the CIA.

The company also has a small London
office in Covent Garden, and it plans to move to a larger,
930m2 space elsewhere in central London this year. Palantir's
main UK customers are part of "Five Eyes", an international
partnership comprising intelligence and security services from the
UK, the US, Canada, Australia and New Zealand. The member countries
share intelligence with one another that's particularly focused on
cyber security and signals intelligence -- intercepted phone-calls,
emails and other electronic messages. The British "Eye" is the
Secret Intelligence Service (MI6) and the Government Communications
Headquarters (GCHQ), the signals-intelligence
agency.

Palantir is extraordinarily
tight-lipped about its UK customers, much more so than about its
American ones. (British secrecylaws are
much stricter and more readily enforced than in the US.) According
to sources, Palantir has few or no customers in domestic law
enforcement, such as Scotland Yard. However, because British
agencies routinely share foreign and domestic information with one
another, there are domestic security organisations in the UK
receiving the benefit of Palantir without having actually used the
software themselves.

Asher Sinensky, who's in charge of
Palantir's UK business, says the British government will be using
Palantir during the Olympics. "There will be half a million more
people in London every day during the games," he says. "The
transportation infrastructure will be swept in ways that it hasn't
before." He says there's also "a lot of concern" among British
officials about defending borders and controlling who comes in and
out of the country. "These outsiders, we don't have the same record
of their interactions with people as we do with [British] citizens
and others we've been tracking."

Joe Pugliese

Palantir was launched in 2004 by Alex Karp, a
financial adviser with a PhD in neoclassical social theory and no
experience running a technology company, and Peter Thiel, a
billionaire venture capitalist who'd helped start PayPal and was an
early investor in Facebook. Karp, a self-described progressive,
knew Thiel, a prominent libertarian, from their days at Stanford
Law School. After 9/11, Karp had reconnected with Thiel, who had
the idea that Silicon Valley should do something to improve
national security and secure civil liberties. Karp, who is tall and
thin with a bushy crop of hair, has none of the swagger of so many
Silicon Valley entrepreneurs. He's a restless academic, more at
home in a seminar than a pitch meeting. During
an interview at his office in Tysons Corner, he stands
up, paces the room for a few moments, sits down, gets up again and
starts sketching out diagrams and graphs on a white board to
explain how Palantir works.

They and three other cofounders
wanted to use PayPal's fraud-detection technology as the model for
a new counterterrorism software, which would be used by analysts to
crunch huge amounts of data.

The parallels between moving money
and fighting al-Qaeda might not be immediately obvious, but the
ascendancy of PayPal, which was founded in 1998, was largely
because of its ability to prevent criminals from stealing its
customers' money. Engineers designed an algorithm that let human
fraud experts -- many of whom were former law-enforcement officers
-- quickly sift through transaction data, look into the transaction
network and map out connections among suspected criminals. That
approach made PayPal the world's most trusted system for online
payments. In 2002, the company was acquired by eBay for £964
million. "The bread and butter of PayPal was to look at a
transaction and to know if it came from a bad IP address," says Bob
McGrew, Palantir's director of engineering. That same approach
became the heart of Palantir, which Karp says is an "attribution"
software, meaning it's used for finding the people behind pieces of
data.

Palantir wanted to beat terrorists the way PayPal
beat Russian criminal gangs. Thiel, who's something of a policy
dilettante, also thought the idea could make a lot of money.
Palantir is considered one of the most valuable startups in the US
-- VC firm Globespan
Capital Partners estimates Palantir's market valuation to be as
high as £2.5 billion. A top executive at JPMorgan Chase says it's
poised to do for information locked inside organisations what
Google did for information on the web.

Before Karp and Thiel ever signed their first
contract, they sought counsel from some of the most important
national-security experts in Washington. Not long after the company
was officially formed, in 2004, the two cofounders met with John
Poindexter, a former national-security adviser to Ronald Reagan, at
the home of Richard Perle, who was chairman of the Defense Policy Board, a group of influential Pentagon advisers,
in 2001.

"I told them I thought they had an
interesting idea," says Poindexter, who, from 2002 to 2003, ran a
Defense Department initiative called Total
Information Awareness, which bore striking similarities to
Palantir's approach to data analysis. The programme was shut down
following outcries from privacy activists -- TIA proposed to
mine not just government intelligence databases, but privately held
records such as credit-card transactions, email and phone
records.

Experts such as Poindexter helped Palantir open
doors. In a short time, the company has assembled a legion of
advocates from the most influential strata of government. Karp
counts former CIA director George Tenet as a friend; he says the
same about Tenet's employer, Herb Allen, who runs the enigmatic
investment bank Allen
& Co, a Palantir investor. And another top adviser, Bryan
Cunningham, was a CIA intelligence officer and a senior staffer to
former secretary of state Condoleezza Rice.

Early prospective investors were
sceptical of Palantir. Venture-capital firms were looking
for the next Facebook or Google -- most presumed it would come in
the form of a consumer technology, probably a new social-media
site. No one wanted to back an expensive software platform for
large organisations. The Palantir founders met several
venture-capital firms, and all turned them down. "We believed it
would work," Karp says. "No one else did."

Potential investors were also leery
of Washington. "The government was unpopular in Silicon Valley,"
Karp says. At the end of one failed pitch meeting, an investor
who'd turned him down said there was a group he should talk to
"that does this kind of thing". The group was In-Q-Tel, the venture-capital arm of
the CIA that was set up in 1999 to bypass the cumbersome government
procurement process and to fund technologies that might be useful
to intelligence agencies.

Palantir got a meeting with
In-Q-Tel's CEO, Gilman Louie, a former computer-game designer.
Stephen Cohen, one of Palantir's founders, was 22 at the time, but
had been writing code since he was a teenager. He spent the next
eight weeks with another cofounder, hammering out a version of
Palantir they could take to the meeting. They worked -- and slept
-- in an office Thiel had used when he founded PayPal.

In-Q-Tel invested a relatively
insignificant amount of money -- about £1.3 million, a small chunk
of the nearly £26 million that Karp says Palantir's investors spent
before the company saw its first dollar in revenue. But it led to a
meeting with another interested backer, the venture wing of Reed
Elsevier, the publishing and information conglomerate. One of
its partners saw Karp give a presentation at an In-Q-Tel meeting,
and was so impressed that he invested a few million dollars.
Crucially, In-Q-Tel put Palantir's founders in the room with
frontline US intelligence analysts, the people they hoped would use
their product. The analysts gave Palantir the software equivalent
of a test drive. "They'd say, 'I love that, I hate that,'" Karp
explains.

The founders spent the next three years flying to
Washington, taking notes and then returning to Palo Alto to tweak
the software. Cohen says he was getting "most of my calories from
Red Bull".

Karp estimates that he and Cohen had more than 300
meetings with likely users, people far down the government
hierarchy. The Silicon Valley techies found themselves deep in an
unfamiliar culture: some people introduced themselves only by their
first names and refused to say where they worked in the
government.

While shuttling between the coasts, Palantir's
founders discovered that intelligence analysts wanted a way to
search their own databases and to know what their colleagues in
other agencies had available. But just as important, agencies
needed to restrict access, so that only those with the proper
security clearances could, for instance, look at the video of a
drone attack or read a classified interrogation summary. Palantir
developed a method for indexing information so the system would
match up a particular data point with the user's security
clearances. If he didn't have the authority to read it, the
information was unavailable. This technique had the added benefit
of creating an audit trail of what the Palantir users were reading,
whether they'd handled the information properly, and whether they'd
modified it in any way.

It's difficult to overstate the
importance of this security regime. Without such nuanced controls
-- down to the level of a single person or one nugget of
intelligence -- the kinds of collaboration necessary to prevent
terrorist attacks just won't happen. An audit trail like this also
lets analysts check their own prior judgments to see if there was a
flaw in their logic.

Palantir also developed a way to organise data that
spoke to a great yearning in the spy world: the need to quickly
assimilate new information into an unfolding narrative. Once data
is put into Palantir, the software uses a model called "dynamic
ontology" to show how names, places and events relate to one
another. For instance, imagine a suspected terrorist who's being
tracked by MI6 makes contact with someone whom the service hasn't
seen yet. That person's name goes into Palantir's system, and the
entire dossier on the original target changes to account for any
previously unseen connections between the two. The network of
relationships between the target and anyone the new person knows
can be seen as well. Every time an analyst adds a new piece of
data, the picture changes automatically. And this new picture can
be shared with other analysts using the software.

Joe Pugliese

Before it had even landed a customer, Palantir was
given a rare audition with the agency that knew
better than any other about the dangers of misguided analysis.
According to a government official, the CIA allowed Palantir to set
up its software in the agency's counterterrorism centre, the hub of
its global campaign to track down terrorists. The official was
astounded that a little-known company from Silicon Valley was
allowed to place its equipment on a network that pulses with the
most highly classified government intelligence. Palantir didn't
disappoint. The official says the company worked for several months
without pay and convinced the CIA that its technology could do what
it claimed. Yet heading into mid-2008, Palantir still hadn't won a
government contract or earned any revenue. The small number of
investors the company had managed to attract were getting
impatient. Karp had already delayed the release of the software by
a year because he felt it wasn't ready. "I'm not motivated by
money," Karp says. "It's not what gets me out of bed in the
morning."

Patience paid off. At yet another
analyst meeting, Cohen showed a group of more senior government
officials what enhancements the engineering team had made. Out of
the corner of his eye, Cohen claims, he saw two stoic men in
grey-flannel suits turn to each other and, without speaking a word,
give each other high fives.

"At that moment, it was really clear to me: we're
going to have a very, very valuable business," Cohen says. Back in Palo Alto, Palantir moved into a 650m2office.
The founders were about to sign their first contract with a
government agency. Karp won't say whom, and although sources'
accounts conflict, it was likely that the client was the CIA or a
Defense Department group set up to fight improvised explosive
devices and bomb makers. Palantir began recruiting top students
from Stanford and other elite computer-science schools.
It offered the typical tech-employee perks, including free dry
cleaning and three meals a day. On the company intranet, Karp sent
out motivational videos that instructed employees how to talk about
Palantir with customers. Employees nicknamed it
KarpTube.

"The office was like a fraternity
for very smart people," says Tim Su, who worked for two years as a
software engineer.

The cultures of Washington and Palo
Alto found enough common ground that Palantir's initial contract
turned into a second and then a third. Palantir's early business
grew based on word of mouth -- Karp calls it "a rumour mill of
people who'd worked with the product". They called friends in other
agencies and urged them to buy it.

Beyond government, Palantir's business now includes
prominent banks and financial institutions -- such as JPMorgan
Chase and the hedge-fund manager Bridgewater -- and is moving
into healthcare, helping to spot fraud and inefficient spending.
Media reports estimate that the company earned just over $250
million in 2011.

Palantir's corporate ethos sometimes
feels more connected to fantasy than reality. In public remarks,
several employees have said their job is to assist the people "who
are out saving The Shire". It's an allusion to JRR Tolkien's
The Lord of the Rings. The Shire is the home of the
hobbits, who band together with their elf, dwarf and human
compatriots to save the world from the armies of Sauron, the master
of evil. The company is permeated with Tolkien
references.

Its Palo Alto office is known as The
Shire, and the office in Virginia is Rivendell -- the home of the
elves. The London office is Grey Havens, an elvish port. The firm's
name is also from Tolkien -- a palantir is a magical stone that
lets its holder see across great distances. The stone is also used
by Sauron to conduct surveillance as he wages war. It's an apt
allusion: just like Tolkien's palantir, the ends to which the
Palantir software is used depends on who's manipulating it. And
that includes not just Palantir's clients but its own employees,
some of whom have embraced their powerful status as an arm of the
surveillance state.

In the autumn of 2010, Palantir employees partnered
with an ex-US Navy intelligence analyst named Aaron Barr, the new
CEO of HBGary Federal, a
company specialising in identifying computer viruses, on an
ill-conceived project that carried the promise of big money -- but
also a lot of risk. Palantir and HBGary Federal teamed up with a
third intelligence contractor, Berico Technologies,
to provide information on groups and individuals deemed hostile to
the US Chamber of Commerce. The law firm Hunton & Williams
first approached Palantir about the work, which was to include
reconnaissance of various websites and social media in order to
build dossiers on the Chamber's opponents. According to a proposal,
Palantir would "serve as the foundation for all of the data
collection, integration, analysis, and production
efforts".

For its work, Palantir asked to be
paid $1.1 million. Anticipating that its client might balk at such
a price, Matthew Steckman, a Palantir employee in the Washington
office, wrote an email to his teammates urging them to emphasise,
"We are the best money can buy! Damn it feels good to be a
gangsta."

A few days afterwards, the law firm
asked whether it could offer a proposal for another job, this time
targeting the anti-secrecy group WikiLeaks, which at the time was
threatening to release internal records from Bank of America. As
Steckman explained to his team, Bank of America wanted to sue
WikiLeaks and enjoin it from releasing the information.

The US Justice Department, which had
been looking for a way to prosecute WikiLeaks's founder, Julian
Assange, called Bank of America's attorneys and told them to get in
touch with Hunton &
Williams. "Apparently, if they can show that WikiLeaks is
hosting data in certain countries, it will make prosecution
easier," Steckman wrote. Barr said that the Palantir team should
target WikiLeaks's "global following and volunteer staff" as well
as people donating money to the group: "[We] also need to get
people to understand that if they support the organisation we will
come after them. Transaction records are easily identifiable." He
said they should submit fake documents to WikiLeaks and try to
foment distrust among different camps of supporters. Barr also
wanted to launch "cyberattacks" on a server WikiLeaks used in
Sweden in order to "get data" about people who were anonymously
submitting information.

But, in February 2011, an article
appeared in the Financial Times quoting Aaron Barr, who
bragged that he'd been able to penetrate the inner ranks of another
hacker-activist group, Anonymous. The group retaliated by breaking
into Barr's email account and publishing several years' worth of
his correspondence, which included the proposals relating to
Palantir.

For Palantir, a company founded on
the idea that technology should protect personal freedoms, it was a
humiliating revelation: the company risked looking like a
cybermercenary. That image ran counter to the core values of the
company. Karp was apparently unaware of what his subordinates had
been doing. Palantir has what Karp calls a "flat hierarchy":
employees are encouraged to act like entrepreneurs and not to seek
approval for every decision they make. Karp says this structure is
essential to Palantir's success: "No company in the Bay Area is
disruptive with multiple layers of hierarchy."

Palantir placed Steckman on leave,
pending a review of his action. Barr resigned from HBGary Federal.
Karp ended all contacts with HBGary and issued a statement
apologising to "progressive organisations… for any involvement that
we may have had in these matters".
Karp says that Palantir hired the law firm Boies, Schiller &
Flexner to investigate the company's role. It recommended that
Palantir should keep Steckman as an employee, which Karp says that
he did. But the incident begged a question: what's to stop a
government intelligence agency from turning off Palantir's
privacy-protection features and using the software for illicit
purposes? Karp insists that the controls are "very hard to
circumvent" and that it would take a "world-class software team" to
do it.

He doesn't cite an example, but one
agency Palantir employees claim hold it in high regard -- the US
National Security Agency (NSA) -- would have both the skill and the
motivation to modify Palantir for its own ends. The agency employs
the largest and most skilled cadre of software experts in the US
government. And for more than four years after 9/11, it conducted a
secret campaign of electronic surveillance against US citizens that
bypassed federal courts. The NSA also took over many of John
Poindexter's Total Information Awareness programmes after they were
officially shut down, but it rejected one: building
privacy-enhancing technology into computer software.

The cultural distinctions between
employees in Palo Alto and Virginia have become more pronounced.
Palantir is neither of the Valley nor entirely of Washington. It's
a kind of techno-military hybrid. In Virginia, there are
predictable trappings of a startup -- a pool table, Razor scooters,
a well-stocked kitchen -- but there are also employees wearing
desert boots and customers in camouflage fatigues.

Palantir, like so many government contractors, has
installed a Washington-style revolving door. Most of its "embedded
analysts" -- employees who work on high-priority
national-security threats -- are former users from the military and
intelligence community. A job description on Palantir's website
describes the ideal embedded analyst this way: "Although you loathe
the bureaucracy, you have a deeply held belief that a revolution in
intelligence affairs is not only possible, it is imminent. Help us
craft that revolution."

Has Palantir created more of a cult
than a culture? Karp -- whom employees call Dr Karp -- insists
they've built "a culture that's not based on money". Palantir caps
all salaries at $127,000 (£82,000). Employees are compensated with
bonuses and equity stakes, but most of the engineers could make far
larger salaries if they defected to Facebook or Google. If someone
goes to work for Palantir, it's probably because he or she believes
in Palantir and its mission. Despite Palantir's obvious trajectory
toward a public offering of stock in the near future, he says, "I
don't want an IPO. The minute you have it, people wake up and ask,
'How rich am I?'"

Today, some current and former US
government officials say Palantir's star has dimmed in the
intelligence community. They complain that the software has a hard
time analysing very large databases and that it takes a lot of time
on the front end to arrange information in a format Palantir can
use.

Still, Palantir has managed to build
what it claims, and, despite its shortcomings, it is a technology
that has made possible several significant contributions to solving
some of the US's most important national security challenges. "The
contradiction that we wanted to remove was between civil liberties
and fighting terrorism," Karp said at a recent Palantir conference.
"Do we really want to live in a world where everyone sees
everything without any kind of permissions? Solving this problem…
that's a really cool idea."

Security and liberty are competitors
now. That's not a natural condition; it's a product of our time, of
the decisions that we have all made -- or failed to make -- over
the past decade. Could a piece of software allay that uneasy
tension? Perhaps. But as any good student of Tolkien knows, whether
a palantir is used for good or for evil depends on who's holding
the stone.

1 --
Intelligence In 2011, Palantir provided a software platform to
investigators at the Center for Public Integrity, in conjunction
with Georgetown University, to support a three-and-a-half-year
project identifying the kidnappers who killed US journalist Daniel
Pearl in 2002. Palantir's software platform analysed the
investigation data and helped identify the links between the key
individuals involved in the crime.

2 -- Defence Palantir provides a large selection of tools for government
defence analysts and soldiers to improve their access to
battlefield intelligence. One example is their "human terrain"
platform, which stores photos, notes and summaries of key
political, ethnic and economic leaders and trends in a war zone, so
that shuffled units don't have to constantly relearn the
information.

3 -- Disaster
relief After the 2010 Haiti earthquake, Palantir developed a free,
open database, including names and locations of collapsed
buildings, Internally Displaced People camps and SMS messages
within specific administrative sectors. Also, a Palantir analyst
spent a week examining federal government spending associated with
Hurricane Katrina, using geospatial, temporal and relational
analysis.

4 -- Finance Using data from the Regulatory Data Corporation, Palantir
has helped US banks to crunch the numbers to spot patterns of
fraudulent mortgage loans perpetrated by organised-crime rings and
street gangs. Their platform can also sniff out individual fake
transactions and identify corporate and human networks operating
under the radar.

5 --
Healthcare The Palantir Health platform is used by US government
health authorities, such as Medicare and Medicaid, to analyse
electronic health records. The software enables Palantir to
reconstruct a history of interactions between patients, doctors,
insurers and others involved, in order to find subtle but
useful patterns.