Share this story

Further Reading

It is no secret that Tesla's Autopilot project is struggling. Last summer, we covered a report that Tesla was bleeding talent from its Autopilot division. Tesla Autopilot head Sterling Anderson quit Tesla at the end of 2016. His replacement was Chris Lattner, who had previously created the Swift programming language at Apple. But Lattner only lasted six months before departing last June.

Keller was a well-known chip designer at AMD before he was recruited to lead Tesla's hardware engineering efforts for Autopilot in 2016. Keller has been working to develop custom silicon for Autopilot, potentially replacing the Nvidia chips being used in today's Tesla vehicles. When Lattner left Tesla last June, Keller was given broader authority over the Autopilot program as a whole.

Keller's departure comes just weeks after the death of Walter Huang, a driver whose Model X vehicle slammed into a concrete lane divider in Mountain View, California. Tesla has said Autopilot was engaged at the time of the crash. Tesla has since gotten into public feuds with both Huang's family and the National Transportation Safety Board, the federal agency investigating the crash.

"Today is Jim Keller’s last day at Tesla, where he has overseen low-voltage hardware, Autopilot software and infotainment," Tesla said in a statement to Electrek. "Prior to joining Tesla, Jim’s core passion was microprocessor engineering, and he’s now joining a company where he’ll be able to once again focus on this exclusively."

Intel announced Thursday that it was hiring Keller as a senior vice president. Keller will "will lead the company’s silicon engineering, which encompasses system-on-chip (SoC) development and integration," an Intel press release said.

Tesla has a huge Autopilot problem

We don't know why Keller decided to leave. What we do know, however, is that he faced a hugely challenging situation at Tesla.

For the last 18 months, Tesla has been charging customers $3,000 for a "full self-driving" option for Autopilot. Tesla has promised customers that its current hardware—which includes radar and a bunch of cameras but no lidar—will be sufficient to achieve full self-driving capabilities. And since June 2016, Elon Musk has been predicting that full self-driving capabilities would be available within the next two years.

But nearly two years after Musk's prediction, Tesla doesn't seem to be close to delivering full self-driving capabilities to customers who paid for the feature. And there's a real question whether it's even possible to deliver full self-driving capabilities with the hardware Tesla has shipped since 2016.

Every other major company working on fully self-driving technology is using lidar as a key part of its sensor suite. Musk has repeatedly dismissed lidar as a "crutch," but it seems likely companies like Waymo and GM know what they're doing. Tesla's cars have fewer hardware redundancies than Waymo's and GM's cars, raising potential safety concerns if key components fail. And it's not known if the Nvidia processors on current Tesla cars are even powerful enough for full self-driving hardware.

Indeed, reporting from TheWall Street Journal suggests that Sterling Anderson, the head of the Autopilot program when Tesla started selling the "full self-driving" feature, wasn't sure the hardware would be sufficient for full self-driving capabilities. "The announcement shook up some engineers, because they believed that the product that was released wasn’t designed to be self-driving," the Journal said. When Anderson was asked about the move by an employee, he reportedly responded that "this was Elon’s decision."

The decision to market Autopilot as a "full self-driving" system "was a factor in the decision by Mr. Anderson and at least two other engineers to leave the company," the Journal reported.

Since Anderson left, the project has faced a steady talent drain. In its piece last August, the Journal noted that the team "lost at least 10 engineers and four top managers." That included Anderson's successor, Chris Lattner, who lasted only six months.

And Lattner wasn't subtle when he left Tesla last summer.

Turns out that Tesla isn't a good fit for me after all. I'm interested to hear about interesting roles for a seasoned engineering leader!

Why running the Autopilot team is so hard

Enlarge/ Waymo is building a driverless taxi service, a business model that might be better suited to fully self-driving vehicles.

Waymo

According to Electrek, Tesla is planning to split Keller's role into two jobs. Chip designer Pete Bannon will succeed Keller as head of Autopilot hardware efforts. Meanwhile, Andrej Karpathy, the head of Autopilot's machine-vision efforts, will be promoted to lead all Autopilot software efforts.

On the one hand, the Autopilot team is facing pressure from regulators and safety advocates to more clearly position Autopilot as a driver-assistance feature. That could mean further limiting how long Autopilot will operate without a driver's hands on the wheel—or reducing the number of warnings a driver gets before the technology shuts off. It could mean adding new techniques for making sure the driver is paying attention: for example, a driver-facing camera. And it could mean limiting operation of Autopilot to only roads where Tesla is confident it will be safe.

At the same time, the Autopilot team faces pressure from Musk—and from customers who paid $3,000 for full self-driving capabilities—to expand Autopilot's capabilities and ultimately enable hands-free operation on all roads. Ideally, they'd figure out how to do that without requiring lidar sensors or other expensive upgrades.

The problem, of course, is that this might not be possible.

Indeed, Google—now Waymo—concluded years ago that the evolutionary approach Tesla is currently pursuing didn't make sense. Engineers observed that test drivers were too quick to trust early versions of its self-driving technology even when they were explicitly told to keep their hands on the wheel and their eyes on the road. So Google scrapped plans to market a driver-assistance technology and instead focused on building fully self-driving cars where a customer would never have to touch the wheel.

Further Reading

Waymo has focused on building an autonomous taxi service because that allows the company to roll its service out gradually, city by city. Waymo plans to start in Phoenix, where there are pristine roads and perfect weather 99 percent of the time. Over time, as its software becomes more capable, it will expand service to areas with more challenging situations—like snow or high-density urban environments. This strategy means that Waymo can add or replace sensors and other hardware to its cars as it learns what's required for full self-driving capabilities.

But that strategy isn't a good fit for a company like Tesla that's based on selling cars directly to customers. Customers expect their cars to work everywhere, not just in one particular metropolitan area or only in favorable weather conditions. Tesla doesn't have billions of dollars to spend mapping city streets and exhaustively testing the software with safety drivers. And customers who already paid for full self-driving mode aren't going to want to pay thousands of dollars extra to add lidar or additional processing power.

In short, Bannon and Karpathy have a challenging job ahead of them. If Elon Musk doesn't want to keep losing Autopilot leaders, he might want to consider modifying Tesla's Autopilot strategy to make sure that the job isn't impossible.

Promoted Comments

Keller typically builds up a team and lays the foundation for the hardware design, then after a few years moves on to another project or company. Everyone was claiming doom at AMD when it was announced he was leaving, but Ryzen has proven to be fine.

I doubt he wanted to head the software side of Tesla's autopilot any longer than he had too, but he has likely laid out the groundwork for Tesla to build their own custom hardware that we will probably see the fruits of in a year or two. Now if it's true he's off to Intel we'll have to see what he'll be up to next but it's bound to be interesting.

This is the guy that has worked on DEC Alpha's, K7-K8, A4 and A5, and Zen. His longest stint prior to AMD was at PA Semi which was acquired by Apple where he stayed 4 years after they acquired it.

"markings that imitate lane markers but lead to a solid divider" as a corner case that requires special workarounds? That is nuts. A stationary object directly in front of the car is not a corner case, it is a primary case. And I don't care if white lines do go right up to it, the imminent collision should take massive f*cking priority.

The current software is designed similar to other level 2 software and filters out all stationary objects at highway speeds. There isn't any level 2 software that will not crash into that barrier if there are markings that imitate standard lane markings leading into it.

Can we please dispense with the bullshit claim that lane markings "lead into" this barrier?

It looks like a lane further down the road, and ought to be painted in or otherwise indicated to be non-lane in character, but the Tesla necessarily crossed a solid white line to reach the barrier. You aren't supposed to cross solid white lines. You certainly aren't "lead" across them.

Google spent years and millions of miles developing their self driving tech. And it is still not ready. Why does Tesla think their, much shorter and smaller scale effort, is ready to be sold and placed in cars sold to consumers?

Google spent years and millions of miles developing their self driving tech. And it is still not ready. Why does Tesla think their, much shorter and smaller scale effort, is ready to be sold and placed in cars sold to consumers?

'member George Hotz was going to have his nonsense thing on the market by now?

Google spent years and millions of miles developing their self driving tech. And it is still not ready. Why does Tesla think their, much shorter and smaller scale effort, is ready to be sold and placed in cars sold to consumers?

That article says that Karpathy became head of "Autopilot Vision," which was part of Lattner's job but not the same as being boss of the overall Autopilot project.

From the article: "Andrej will work closely with Jim Keller, who now has overall responsibility for Autopilot hardware and software."

It's true that the job duties have gotten shifted around somewhat with each transition, but Keller came head of the overall Autopilot project after Lattner's departure, so I think it's fair to treat him as the main successor.

Google spent years and millions of miles developing their self driving tech. And it is still not ready. Why does Tesla think their, much shorter and smaller scale effort, is ready to be sold and placed in cars sold to consumers?

Google spent years and millions of miles developing their self driving tech. And it is still not ready. Why does Tesla think their, much shorter and smaller scale effort, is ready to be sold and placed in cars sold to consumers?

Spoiler: show

Is Tesla out of touch for thinking they can offer a completely autonomous driving package as a retroactive vehicle upgrade for a few thousand dollars with a hot-mess of software? No, it's every other company who is out of touch!

Google spent years and millions of miles developing their self driving tech. And it is still not ready. Why does Tesla think their, much shorter and smaller scale effort, is ready to be sold and placed in cars sold to consumers?

Elon’s hubris has no bounds.

This is true, and in most circumstances, it has been a good thing. Time will tell whether this is the exception.

Google spent years and millions of miles developing their self driving tech. And it is still not ready. Why does Tesla think their, much shorter and smaller scale effort, is ready to be sold and placed in cars sold to consumers?

Elon’s hubris has no bounds.

Sometimes, in the course of business, you find yourself having to sell somebody a product that doesn't exist yet... and which has never existed before. There's some risk to this.

When you're General Electric promising new jet engines to Boeing, or Intel promising new chips to Apple, or whatever, you're in a well-worn saddle. You're trying to do what's never been done, and you might fail, but you have one or two big customers who know this. Your customers have been in this position before, understand that failures happen sometimes, and you've got robust contractual language to handle what might happen if you fall short of your goals. Perhaps most importantly, everybody is making carefully considered decisions rather than acting on impulse and feeling and emotion.

I try to be measured in my criticism, but selling regular Joes cars that you promise will be able to drive themselves with the hardware already on-board, when nobody's ever built a successful self-driving car before, was unspeakably daring. It brought a bunch of money in the door. It might also have brought doom.

Google spent years and millions of miles developing their self driving tech. And it is still not ready. Why does Tesla think their, much shorter and smaller scale effort, is ready to be sold and placed in cars sold to consumers?

It would help if Tesla would stop calling their driver-assistance system "autopilot". It has literally nothing to do with an autopilot feature, as the term is understood!

This is a hole that Tesla is digging for themselves.

As I understand it, it works pretty much exactly like how existing auto pilot systems work in aviation. They are provided a heading and beyond that the system will make steering and throttle adjustments to maintain that heading. The pilot is still expected to be present and ready to take over at any time and the system is not expected to fly the plane without a pilot.

I saw at least one article saying Jim Keller was headed to Intel. That's a pretty huge pickup for Intel. It's a big, but maybe not unexpected, loss for Tesla. Keller has been known to move companies every few years. It'll be interesting to see the mark he leaves on a company that big in just a few years.

It would help if Tesla would stop calling their driver-assistance system "autopilot". It has literally nothing to do with an autopilot feature, as the term is understood!

This is a hole that Tesla is digging for themselves.

As I understand it, it works pretty much exactly like how existing auto pilot systems work in aviation. They are provided a heading and beyond that the system will make steering and throttle adjustments to maintain that heading. The pilot is still expected to be present and ready to take over at any time and the system is not expected to fly the plane without a pilot.

Do those aviation systems market to consumers, though? It's one thing to have a technical definition for autopilot that's understood by actual pilots, but it's another to have a marketing name for consumers that has "Full Self Driving Hardware on All Cars" as well as:

Quote:

All Tesla vehicles produced in our factory, including Model 3, have the hardware needed for full self-driving capability at a safety level substantially greater than that of a human driver.

It would help if Tesla would stop calling their driver-assistance system "autopilot". It has literally nothing to do with an autopilot feature, as the term is understood!

This is a hole that Tesla is digging for themselves.

As I understand it, it works pretty much exactly like how existing auto pilot systems work in aviation. They are provided a heading and beyond that the system will make steering and throttle adjustments to maintain that heading. The pilot is still expected to be present and ready to take over at any time and the system is not expected to fly the plane without a pilot.

True, but the likelihood of contact with another vehicle/tree/person, etc. is much lower at 30,000 feet...

I saw at least one article saying Jim Keller was headed to Intel. That's a pretty huge pickup for Intel. It's a big, but maybe not unexpected, loss for Tesla. Keller has been known to move companies every few years. It'll be interesting to see the mark he leaves on a company that big in just a few years.

Yes, that's what Electrek reported. I haven't been able to independently confirm it though so I left that detail out of the story.

It would help if Tesla would stop calling their driver-assistance system "autopilot". It has literally nothing to do with an autopilot feature, as the term is understood!

This is a hole that Tesla is digging for themselves.

As I understand it, it works pretty much exactly like how existing auto pilot systems work in aviation. They are provided a heading and beyond that the system will make steering and throttle adjustments to maintain that heading. The pilot is still expected to be present and ready to take over at any time and the system is not expected to fly the plane without a pilot.

You are correct. I should be clearer in what I was saying.

To the 'average' person - including myself, someone who hasn't so much as touched a flight simulator - autopilot means...well, that the vehicle pilots itself. You enter a destination and the vehicle delivers you to it. I think this is the understanding of the term 'autopilot' that is being employed catastrophically. Obviously, your explanation has remedied that for me, but still.

I think that Tesla should stop calling their autopilot an autopilot feature due to the expectations it generates. This is one of those times when calling the feature something accurate still manages to set the wrong set of impressions, due to the difference between how qualified, informed individuals understand a term, and how the unwashed masses (again, I'm including myself in this description) understand that same term.

Put differently, when marketing to the masses, there's wisdom in taking the lowest common denominator into account.

It would help if Tesla would stop calling their driver-assistance system "autopilot". It has literally nothing to do with an autopilot feature, as the term is understood!

This is a hole that Tesla is digging for themselves.

As I understand it, it works pretty much exactly like how existing auto pilot systems work in aviation. They are provided a heading and beyond that the system will make steering and throttle adjustments to maintain that heading. The pilot is still expected to be present and ready to take over at any time and the system is not expected to fly the plane without a pilot.

True, but the likelihood of contact with another vehicle/tree/person, etc. is much lower at 30,000 feet...

I don't think they should have taken any deposits on this added function. They were too far away from production (and probably still are). It is one thing to aspire to vaporware and quite another to sell it.

I have no idea why people are buying FSD; that one seems like throwing money away. And yes, Autopilot was an extremely poor choice of names.

But, Autopilot's ability to drive has significantly improved in the last couple months according to what I've read from Tesla owners. That doesn't really fit with claims of a brain drain in the Autopilot group.

It would help if Tesla would stop calling their driver-assistance system "autopilot". It has literally nothing to do with an autopilot feature, as the term is understood!

This is a hole that Tesla is digging for themselves.

As I understand it, it works pretty much exactly like how existing auto pilot systems work in aviation. They are provided a heading and beyond that the system will make steering and throttle adjustments to maintain that heading. The pilot is still expected to be present and ready to take over at any time and the system is not expected to fly the plane without a pilot.

True, but the likelihood of contact with another vehicle/tree/person, etc. is much lower at 30,000 feet...

I'm a pilot. Autopilot's job is very very simple. Hold heading, attitude, altitude, or two or all three at the same time. Or more advanced ones follow preplanned waypoints, GPS or INS. If something goes wrong the pilot has literally a minute or more to disconnect the autopilot and fly the plane manually.

Car autopilot has orders of magnitude more complex problems to solve, and reaction time measured in 1-3 seconds. Think of it. Jaywalkers, cyclists, large potholes, construction zones, police directing traffic, school crossing guards holding up the stop sign? Car needs to deal with all of that.

It would help if Tesla would stop calling their driver-assistance system "autopilot". It has literally nothing to do with an autopilot feature, as the term is understood!

This is a hole that Tesla is digging for themselves.

As I understand it, it works pretty much exactly like how existing auto pilot systems work in aviation. They are provided a heading and beyond that the system will make steering and throttle adjustments to maintain that heading. The pilot is still expected to be present and ready to take over at any time and the system is not expected to fly the plane without a pilot.

You are correct. I should be clearer in what I was saying.

To the 'average' person - including myself, someone who hasn't so much as touched a flight simulator - autopilot means...well, that the vehicle pilots itself. You enter a destination and the vehicle delivers you to it. I think this is the understanding of the term 'autopilot' that is being employed catastrophically. Obviously, your explanation has remedied that for me, but still.

I think that Tesla should stop calling their autopilot an autopilot feature due to the expectations it generates. This is one of those times when calling the feature something accurate still manages to set the wrong set of impressions, due to the difference between how qualified, informed individuals understand a term, and how the unwashed masses (again, I'm including myself in this description) understand that same term.

Put differently, when marketing to the masses, there's wisdom in taking the lowest common denominator into account.

I think part of the problem is Tesla's marketing. They've sold the auto pilot system as a fully autonomous driving system when it's far from it.

If they were clear that auto pilot means: throttle control + maintain a heading + collision detection/avoidance; I think they'd be having a lot less issues.

GM is doing it right by calling their technology Super Cruise because that sets the right expectations. Of course I'd bet you money that they're having a harder time selling it to customers than Tesla and their auto pilot with its false promises.

Both Lattner and Keller worked successfully at Apple under Steve Jobs. Steve Jobs, in case you didn't know, was not a guy with low expectations, or a person who pulled his punches when offering criticism, or a guy who refrained from micromanaging important projects.

This says to me that the downsides of working for Musk are far greater than the downsides of working with Jobs, but without enough upside to make it worthwhile.

I think the evidence is very clear -- Tesla Autopilot is a complete failure and that isn't changing anytime soon.

It would help if Tesla would stop calling their driver-assistance system "autopilot". It has literally nothing to do with an autopilot feature, as the term is understood!

This is a hole that Tesla is digging for themselves.

As I understand it, it works pretty much exactly like how existing auto pilot systems work in aviation. They are provided a heading and beyond that the system will make steering and throttle adjustments to maintain that heading. The pilot is still expected to be present and ready to take over at any time and the system is not expected to fly the plane without a pilot.

True, but the likelihood of contact with another vehicle/tree/person, etc. is much lower at 30,000 feet...

Aviation auto pilot systems still have collision alert and avoidance.

Whoop whoop. Ter-rain ahead! Ter-rain ahead. Whoop-whoop!

What's really embarrassing is hearing that while landing. Forgot to lower the landing gear.

Google spent years and millions of miles developing their self driving tech. And it is still not ready. Why does Tesla think their, much shorter and smaller scale effort, is ready to be sold and placed in cars sold to consumers?

But it's just training data for their future autonomous system, whereas Waymo et al's miles are returning feedback about their autonomous system.

Also Musk refuses to consider Li-DAR which would be essential for at least night-time driving (because without it, Autopilot has to mostly rely on RADAR).

However, I would suspect that Musk's resistance to Li-DAR isn't technical... Li-DAR systems are big and not very pretty, and good solid state Li-DAR is some ways away. Also the model of selling a software-upgradable hardware platform (i.e.: the car) doesn't work so well if you need to add expensive equipment for a functionality that not everyone might want.

If Nvidia was rolling out chips twice as powerful as what is currently used by Tesla, how hard would it be to just swap out? Would it be just a chip, or an entire board?Is doubling the processing power going to help much, or do we need a couple more generations in advancements?

This is a clear sign that the numbers aren't there. Producing custom chips is just one vertical too many for a company that will have trouble making ends meet this year. I'm not surprised that someone with his background didn't want to wait out the crisis just on the hope of better days.

And it's another confirmation that this was the major miscalculation by Musk. Autopilot will be the anchor.

While everything else DOES sound really bad, Keller has always been a bit of a nomad and moves on to a new exciting project every few years regardless of where he is at. I'm actually curious to see how he gels with Intel since he's spent the bulk of his career either working with emerging companies/divisions rather than market leaders. Hopefully they'll let him run a skunkworks on a new architecture since he seems fairly allergic to drama.

I saw at least one article saying Jim Keller was headed to Intel. That's a pretty huge pickup for Intel. It's a big, but maybe not unexpected, loss for Tesla. Keller has been known to move companies every few years. It'll be interesting to see the mark he leaves on a company that big in just a few years.

It would help if Tesla would stop calling their driver-assistance system "autopilot". It has literally nothing to do with an autopilot feature, as the term is understood!

This is a hole that Tesla is digging for themselves.

As I understand it, it works pretty much exactly like how existing auto pilot systems work in aviation. They are provided a heading and beyond that the system will make steering and throttle adjustments to maintain that heading. The pilot is still expected to be present and ready to take over at any time and the system is not expected to fly the plane without a pilot.

Do those aviation systems market to consumers, though? It's one thing to have a technical definition for autopilot that's understood by actual pilots, but it's another to have a marketing name for consumers that has "Full Self Driving Hardware on All Cars" as well as:

Quote:

All Tesla vehicles produced in our factory, including Model 3, have the hardware needed for full self-driving capability at a safety level substantially greater than that of a human driver.

I'm not disagreeing with the aviation definition of autopilot, but it's clearly not how Tesla is marketing their feature right now.

I think we more or less agree. I only disagreed initially because the original post said with emphasis that it has "literally nothing" to do with auto pilot systems when in fact it literally fits the definition exactly.

Tesla's marketing here is lying to customers and misleading them. People that might assume an auto pilot is more than just that are being encouraged to believe it's a fully autonomous system.