America’s car industry dismissed the potential of autonomous driving for years as tech companies plunged ahead. Now Detroit is racing to catch up.

Early in 2011, two top engineers for Google traveled together to Detroit on what amounted to a diplomatic mission. They had just spent 18 months on a top-secret project called Chauffeur: the development of a car that could drive itself over 10 different 100-mile routes on public roads. Now they were looking for a partner to carry the project forward. “The idea was, if you’re going to make self-driving cars, you have to work with a car company,” recalls Chris Urmson, who made the trip with fellow engineer Anthony Levandowski. “Maybe they’ll sell us cars to build a fleet. Maybe we’re going to be retrofitting our stuff onto their cars to sell.”

But they couldn’t find any takers. In meetings with a prime parts supplier to the car makers and then with the senior leadership of a major auto company, the pair gave presentations on their vehicle’s capabilities, the number of miles it had driven and the broad strokes of how their self-driving software saw the road. The reaction, they say, was utter disinterest—and dismay that they were experimenting on public roads rather than on a test track. “Self-driving technology didn’t make sense to them,” Mr. Urmson says. “And it seemed so far out of the playbook that it wasn’t even addressable.” As they headed back to the airport, Mr. Urmson said to his partner, “Well, I guess we’re not working with those guys.”

Today, self-driving technology is spurring Detroit’s biggest deals. The SoftBank Vision Fund announced plans this spring to invest $2.25 billion in GM’s self-driving subsidiary, Cruise Automation—with GM committing another $1.1 billion of its own money. In July, Ford announced plans to invest $4 billion in its own autonomous car startup. This month, Morgan Stanley analyst Brian Nowak speculated that the company born in late 2016 from Google’s self-driving car team, known as Waymo, could be worth $175 billion—40% more than the combined market capitalization of GM, Ford and Fiat-Chrysler.

Autonomous technology is so hot today that it’s easy to forget Detroit’s initial resistance. Why was the center of the American car industry so dismissive back in 2011, and how did it come around to the fast-approaching revolution in car travel?

When I joined Chauffeur as a consultant at the beginning of 2011, I became the team’s first insider with experience as a car-company executive. Chauffeur was trying to reinvent the auto industry, something I’d been trying to do in my own way as the chief of research and development at General Motors for a decade, including work on autonomous vehicles. The closest we got during my tenure at GM was a joint project developing a two-person autonomous pod with Doug Field, then of Segway (and now making news for his leap from Tesla to Apple’s autonomous car project). Our prototype made its debut the week after GM CEO Rick Wagoner’s 2009 resignation and left little public mark as my employer lurched toward bankruptcy.

Amid the worst recession that the auto makers had ever faced, I understood why they would steer clear of ushering in a technology poised to make them irrelevant. But the deeper reason the auto companies were late to the revolution is that they mistakenly believed that their business was manufacturing and selling cars. They failed to see that their success had always been based on something more fundamental: helping people to get from one place to another.

Auto executives initially dismissed self-driving cars in part because they didn’t understand the full potential of digital technology. But it was also because they were primarily focused on delivering attractive vehicles to dealer showrooms rather than on providing compelling transportation experiences to customers. Detroit was held captive by a century-old business model.

The disconnect on autonomous cars was not only the fault of the auto industry. Early on, I was struck by the disregard the Chauffeur team displayed toward Detroit. Chauffeur’s engineers asked me about Detroit’s product development cycles. When I told them that auto companies typically took around three years to develop a new car, they were astonished. Three years? What on earth took so long?

The engineers’ general attitude was that the auto companies were lazy, out of touch. They didn’t know how to do innovation—at least, not the kind that might spur social disruption, the kind in which Silicon Valley liked to think it specialized. Many on Chauffeur’s team believed that Henry Ford was a remarkable innovator but that somewhere along the line, that spirit had withered in Detroit.

The low point in the mutual aversion between Silicon Valley and Detroit came more than a year after Mr. Urmson and Mr. Levandowski’s failed trip, when Mr. Urmson invited a senior GM representative to come out to Google headquarters in Mountain View, Calif. to take a ride in one of Chauffeur’s self-driving vehicles. The executive spent the ride sharing his negative take on the experience. “I’m sorry,” said the guy, according to Mr. Urmson. “But I just don’t get the point. “

Things began to change with the next stage of development, the combination of driverless tech with another new disruption: on-demand ridesharing services. Mr. Urmson initially revealed the idea for a new concept vehicle called Firefly at an all-hands Chauffeur meeting in December 2012.

Ridesharing was a hot trend in tech investing that year. An entrepreneur named Sunil Paul had begun arranging rides in San Francisco via a mobile app called Sidecar, spurring a pair of mobility entrepreneurs named Logan Green and John Zimmer to roll out their own anyone-to-anyone ride-sharing app, which they referred to as Lyft. Uber followed soon after with UberX.

Chauffeur’s chief engineer told the team that he wanted to pursue on-demand mobility as a business model—essentially, a driverless version of Uber or Lyft. To do that, Mr. Urmson wanted Chauffeur to design a vehicle for the express use of ride-sharing services. He envisioned a world of driverless taxis zipping about cities, picking up passengers, providing rides, then setting off on the next call—what was referred to inside Chauffeur as “transportation as a service.”

I was thrilled with Mr. Urmson’s new project. Research that I had recently led at Columbia University and presented to Chauffeur concluded that this new model could provide mobility experiences better than personal car ownership. The point-to-point travel would be just as fast while allowing riders to forget about inconveniences such as finding a parking space and refueling.

Our analysis later showed that the new business could do all that while saving people most of what they paid for trips in gas-powered, personally owned vehicles, costing them just 20 cents a mile on average compared with a 65-cent average for drives today. (Other studies have found similar costs and savings.) This didn’t even count another 85 cents’ worth per mile of productive time lost while driving, which they could use for other things while traveling as passengers. If just 10% of driving were diverted to this model, the analysis suggested, it could save on the order of $150 billion a year in operating costs and about another $250 billion in lost driver productivity.

The new project set Chauffeur on a fascinating design exercise. What should a driverless vehicle built expressly for the new mobility services look like? The Firefly designers came up with a simple, clean and fun aesthetic. Because Mr. Urmson hoped it would liberate transportation for those who couldn’t drive—the elderly or disabled, for example—the vehicle needed to be easy to enter and exit. To achieve that, the floor would be flat and not too far from the ground. When the conversation turned to the placement and feel of traditional car components, the team made a radical decision: no steering wheel. Why did the Google mobility pod need one?

In May 2014, Sergey Brin unveiled Firefly publicly. By October, Uber CEO Travis Kalanick realized how urgent it was for Uber to develop its own strategy, according to court documents from this year’s trade-secrets lawsuit between Uber and the Google spinoff Waymo, where I remain a consultant. Mr. Kalanick knew that a ridesharing business that operated driverlessly could provide its services for much less than a human-operated rival; the human driver accounted for a reported 70% to 90% of Uber’s cost per mile. Google had earlier invested $258 million in Uber and placed its chief legal officer, David Drummond, on Uber’s board of directors. After a board meeting, according to Mr. Kalanick, Mr. Drummond told him that Google was intending to compete with Uber in the ride-sharing space, and they agreed that he should recuse himself from the Uber board.

Mr. Holden identified the world’s single greatest concentration of self-driving brainpower outside of Mountain View: Carnegie Mellon University’s National Robotics Engineering Center in Pittsburgh, Mr. Urmson’s onetime employer. According to The Wall Street Journal’s reporting, Uber offered compensation packages that included signing bonuses in the hundreds of thousands of dollars and salaries at least double what the scientists and engineers had made at NREC. All told, 40 NREC staff would leave. Uber essentially gutted the place. “I’ve never seen anything like it,” marveled one Carnegie Mellon observer. “People have been complaining for years that no one understands how important this technology is. Then Uber came in and people were like, ‘Wow, this thing is real.’”

Uber’s mass hiring of NREC’s self-driving talent created an enormous amount of discussion in the auto industry. News stories about the deal pinged around the email accounts of Detroit executives. The move amounted to a high-stakes endorsement of self-driving technology, and it triggered a stampede in the industry.

Uber would soon become more valuable than General Motors. Google was one of the world’s most valuable companies (there were quarters when it could have bought GM outright with its cash reserves). The fact that they were both gunning hard to provide driverless mobility finally helped to convince the auto industry that it had to get serious about autonomous cars.

Renault-Nissan CEO Carlos Ghosn announced plans to sell ten new autonomous vehicles before 2020. Toyota’s Akio Toyoda, who had long opposed the technology, announced a plan to invest a billion dollars to start a 200-researcher artificial intelligence lab in Silicon Valley and promised to have vehicles driving themselves on highways by 2020.

Perhaps the biggest and most startling about-face came from GM. “I believe we’ll see more change in the automotive industry in the next five to ten years than in the past fifty,” said GM CEO Mary Barra in a December 2015 essay published on LinkedIn. “I have committed that we will lead the transformation of our industry.” The essay’s headline proclaimed 2016 as “the year Detroit takes on Silicon Valley.”

The next month, the company invested $500 million in the ride-hailing company Lyft. It launched a Zipcar competitor, Maven. And that March, GM announced its purchase, for $581 million plus incentives, of a forty-employee Silicon Valley startup called Cruise Automation, which had begun its self-driving development more than four years after the founding of Google’s self-driving car project.

As for Waymo, the company retired its groundbreaking Firefly last summer to focus on what it had set out to explore in the first place: installing software for autonomous driving in car makers’ mass-produced vehicles. This spring, it signed agreements for such projects with Fiat Chrysler and Tata Motors’ Jaguar brand.

For years, the technologists working on self-driving cars were frustrated that society at large and the auto industry in particular were either unwilling or unable to understand what was possible. Few of us feel that way today. This last decade has been a learning process for everyone. We’ve all grown and changed our minds about how the new technology will develop.

And the learning has happened on all sides. Detroit has been forced to become more receptive to new ideas. Silicon Valley has come to recognize that innovation, particularly the world-changing kind, is a difficult thing to hurry. It requires great leaps of deduction, constancy of purpose and the discipline to let ideas mature until they’re ready for the public.

One of the most important decisions on Chauffeur’s road to commercialization was who to select to run the company. To fill the CEO position, Larry Page and Sergey Brin tapped former Hyundai Motor America president John Krafcik. The hiring of an auto executive to lead the company that would become Waymo was a remarkable moment in the history of self-driving cars.

The Chauffeur team had long been untroubled by its lack of auto industry experience. In many instances, ignorance of the way Detroit did things was considered an asset. But Mr. Krafcik was an auto industry guy, and his installment as Waymo’s leader was, to my mind, the first acknowledgment that this quest didn’t have to pit Silicon Valley against Detroit—that both sides had expertise to contribute to bringing about the great disruption ahead in mobility. Mr. Krafcik’s hiring was a savvy concession on the part of Google’s founders that maybe, just maybe, the two sides needed one another.