Big companies have dominated defense contracting for more than 60 years, but Palantir is an insurgent. Its founders don’t obey the hierarchical order of the defense-industrial complex, which they think has failed the government and ripped off taxpayers.

They have a point. The files of inspectors general overflow with accounts of how big-name companies over-promised and under-delivered on multibillion-dollar technology “modernization” projects meant to cure the intelligence failures that led to the 9/11 attacks. The FBI and the National Security Agency, both of which wasted billions on custom-built computer systems, offer some of the most dramatic examples of high-priced failure. But the IRS, the Interior Department, and the Pentagon, among others, can claim a place in that undistinguished company.

During a speech in London, where geographic and cultural distance may have put Karp in a more candid mood, he gave a scathing assessment of how big defense contractors—his competition—traditionally do business in Washington.

“People charge you every time you want something fixed, every time you want to build something, before it’s built, after it’s built,” he told the London crowd. “You’re paying by the hour. You’re paying 20 people. You think you’ve hired five.”

Palantir wanted to upend the system. But before Palantir could get to the point that it could offend Washington’s sensibilities, it had to get out of Silicon Valley.

Prospective investors were skeptical. Venture-capital firms were looking for the next Facebook or Google, but most presumed it would come in the form of a consumer technology, probably a new social-media Web site. No one wanted to back an expensive software platform for large organizations. The Palantir founders met with several venture-capital firms, and all turned them down. “We believed it would work,” Karp says. “No one else did.”

Potential investors were also leery of Washington. “The government was unpopular in Silicon Valley,” Karp says.

“Everyone told us we should stay away from Washington because it was corrupt and we didn’t know how to play golf with senators,” says Joe Lonsdale, another Palantir cofounder and former editor-in-chief of the Stanford Review, the libertarian publication Thiel founded as an undergraduate. He recalls that one prospective backer ridiculed Karp for having a PhD in philosophy, not a technical field. Says Lonsdale: “It was kind of a horrible experience.”

According to Karp, at the end of one failed pitch meeting, an investor who’d just turned him down said there was a group he should talk to “that does this kind of thing.” The group was In-Q-Tel, the venture-capital arm of the CIA, which was set up in 1999 to bypass the cumbersome government procurement process and to fund technologies that might be useful to the spy agencies.

In-Q-Tel is a gatekeeper between small technology companies, which usually know Washington only by its location on a map, and the secretive regions of the intelligence community. “If you’re crazy enough to want to work with government, they’re the right people to talk to,” Stephen Cohen, one of the cofounders, told an audience at the Startup School conference held at Stanford in October. Palantir got a meeting with In-Q-Tel’s CEO, Gilman Louie, a former computer-game designer.

“If you guys aren’t able to help us with counterterrorism, you have a real future in video games.”

Cohen was 22 at the time, but had been writing software code since he was a teenager. He spent the next eight weeks with another cofounder hammering out a version of Palantir they could take to the meeting. They worked—and slept—in an office Thiel had used when he founded PayPal.

“Like anytime you’re putting together a prototype really fast,” Cohen said at the conference, “you’ve got to focus a lot on using the fireworks to cover up for the broad strokes on the back end, the pieces that are going to get there eventually.”

Louie was impressed. After the demonstration finished, he told Cohen, “If you guys aren’t able to help us with counterterrorism, you have a real future in video games.”

In-Q-Tel invested a relatively insignificant amount of money—reportedly about $2 million, a small chunk of the nearly $40 million that Karp says Palantir’s investors spent before the company saw its first dollar in revenue. But more crucially than the money itself, In-Q-Tel put Palantir’s founders in the room with frontline intelligence analysts, the people they hoped would one day use their product. The analysts gave Palantir the software equivalent of a test drive. “They’d say, ‘I love that, I hate that,’ ” Karp says.

“We would watch to see how they interacted with it,” Cohen said. “We wanted to know not just what the analysts asked for but really what was happening between the lines.”

During his conference talk, Cohen recalled that at one meeting, his government host introduced him as Mr. Two Weeks. “He can give you anything you want in two weeks,” the host told the assembled analysts. “So ask away.”

The founders spent the next three years flying to Washington, taking notes, and then returning to Palo Alto to tweak the software. Cohen said he was getting “most of my calories from Red Bull.”

Karp estimates that he and Cohen had more than 300 meetings with likely users, people far down in the government’s hierarchy. The Silicon Valley techies found themselves deep into an all-new culture: They encountered some people who introduced themselves only by their first names and refused to say where they worked in the government.

While shuttling between the coasts, Palantir’s founders discovered that intelligence analysts wanted a way to search their own databases and to know what their colleagues in other agencies had available. But just as important, agencies needed to restrict access, so that only those with the proper security clearances could, for instance, look at the video of a drone attack or read a classified interrogation summary. Palantir developed a method for indexing information so the system would match up a particular data point with the user’s security clearances. If he didn’t have the authority to read it, the information was off-limits. This technique had the added benefit of creating an audit trail of what the Palantir users were reading, whether they’d handled the information properly, and whether they’d modified it in any way.

It’s difficult to overstate the importance of this security regime. Absent such nuanced controls, down to the level of a single person or one nugget of intelligence, the kind of collaboration necessary to prevent terrorist attacks won’t happen. An audit trail like this also lets analysts check their own prior judgments to see if there was a flaw in their logic.

Next: “At this moment it was really clear to me: We’re going to have a very, very valuable business.”