A summary of autonomous vehicle testing in 2016 on California’s public roads

The race for self-driving cars began in all earnest in 2015 with Tesla aggressively leading the pack at the end of 2016. Tesla’s CEO, Elon Musk, is of the opinion that current hardware is already sufficient to allow Level 5 automation (full automation) as set by SAE International, an engineering association. KPMG in its 2017 Global Automotive Executive Outlook found that 37% of auto executives rated the self-driving trend as extremely important. The initial front-runner for the technology, Google Auto (now Waymo) has changed strategy from developing a car to developing systems and rather partnering with automakers than building its own car. Other than Tesla, who develop and test on the go with incremental software upgrades. Google and most other contenders are developing solutions in closed environments while being permitted to test on public roads in States such as California, Michigan, and Illinois. By late 2016 a total of 20 companies has received permits from the California Department of Motor Vehicles (CA DMV).

According to the regulations, each permit holder must annually file a disengagement report, reflecting the number of events where a driver essentially has to take over from the vehicle’s autonomous mode to either prevent a traffic incident or where the system fails. These submissions allow the public who is interested what is happening in the automaker’s self-driving test some insights, outside from what the marketing departments feed the media. Although a total of 20 permits have been granted by the end of 2016, only ten companies have conducted public tests or filed reports for the period. Here follows a summary of the reports filed by the various companies for their testing on Califonia’s public roads for the period December 2015 to November 2016. For comparison purposes, we converted the result to indicate the amount of disengagements per 1,000 miles, a measurement used by Waymo in a recent post by its Head of Self-Driving Technology, Dmitri Dolgov.

Disengagements per 1,000 miles

BMW

BOSCH

DELPHI

FORD

GM CRUISE

MERCEDES

NISSAN

TESLA

WAYMO

Cars used

N/A

3

2

2

25

1

5

4

66

Miles

638

983

3,090

590

8,776

673

4,099

532

636,138

Disengagements

1

1,442

179

3

181

336

29

181

124

Per 1,000 miles

1.57

1,467

57.93

5.08

20.62

498.95

7.07

340.23

0.2

Comparing disengagement results per 1,000 miles above shows, bar Tesla’s customers who have clocked over 300 million miles on Autopilot (Level 2 Automation) as at November 2016, it is clear that Waymo is aggressively accelerating its learning and showing the results for it compared to the other brands with projects on Californian roads. One must, however, take into consideration that comparing the different disengagement reports is not truly comparing apples to apples. Ford’s testing, for instance, was only on the stretch of Interstate 10 between Los Angeles and Arizona. The companies also have different strategies. Tesla aims to have level 5 automation as soon as possible while companies such as Ford aims to have autonomous vehicles for ride-hailing and sharing services only from 2021 onwards, influencing the difficulty grade of testing. To further try and unravel each company’s testing program we delve deeper into their submitted reports below.

READ THE BLOG FOR A COMPLETE BREAKDOWN OF DISENGAMENTS REPORTS

BMW Autonomous Disengagement report

Dec-15

Jan-16

Feb-16

Mar-16

Apr-16

May-16

Jun-16

Jul-16

Aug-16

Sep-16

Oct-16

Nov-16

TOTAL

Miles

62

576

638

Disengagements

1

1

Per 1,000 miles

-

-

-

-

1.74

-

-

-

-

-

-

-

1.57

BMW’s two-page report is not very telling. The company only conducted tests over March and April with one disengagement on Highway 101 where the road markings were too faint. Can we deduce from this that the company only tested on highways and freeways?

Bosch Autonomous Disengagement report

Dec-15

Jan-16

Feb-16

Mar-16

Apr-16

May-16

Jun-16

Jul-16

Aug-16

Sep-16

Oct-16

Nov-16

TOTAL

Miles

81

67

52

73

47

298

129

55

19

47

114

983

Disengagements

158

95

152

154

48

239

183

43

31

48

291

1,442

Per 1,000 miles

-

1,945.81

1,420.03

2,911.88

2,100.95

1,021.28

801.48

1,414.22

783.24

1,640.21

1,012.66

2,561.62

1,466.94

The automotive parts company have conducted ongoing tests with three vehicles from the start of 2016, but in total the tests were only over 983 miles. The company tested on interstates, freeways, state highways, public rural roads or streets and parking facilities. Most of Bosch’s disengagements were on interstates and freeways with only two on rural roads or other streets. Bosch lists disengagements as:

Due to weather conditions;

Road surface conditions;

Construction emergencies;

Accidents or collisions; and

Planned test of the technology.

In total, the company classified all its disengagements as “Planned test of Technology.”

Delphi Automotive Autonomous Disengagement report

Dec-15

Jan-16

Feb-16

Mar-16

Apr-16

May-16

Jun-16

Jul-16

Aug-16

Sep-16

Oct-16

Nov-16

TOTAL

Miles

468

255

297

103

48

133

152

319

153

395

577

190

3,090

Disengagements

27

25

13

4

4

7

8

42

12

8

25

4

179

Per 1,000 miles

57.70

98.04

43.73

38.80

83.16

52.71

52.77

131.62

78.28

20.27

43.33

21.03

57.93

Delphi Automotive, like its competitor Bosch, have also conducted ongoing test over the 12 month period but put more miles under its belt, completing testing over 3,090 miles. Delphi tested with two 2014 Audi SQ5’s, one black (1583 miles) and one white (1542.3 miles). Delphi’s conducted tests on US 101, Highway 85, Central Expressway I280 in the San Francisco Bay area and streets in Mountain View and Palo Alto. Delphi indicated that its driver always has one hand on the steering wheel, allowing him/her to execute a disengagement in less than a second for the following scenarios:

Issue

Incidents

Construction Zones

19

To Complete lane change in heavy traffic

62

Emergency vehicles

1

Poor lane markings

10

Precautionary intervention due to heavy pedestrian traffic

27

Precautionary intervention to give extra space for a cyclist

13

Traffic light detection (in poor sun conditions)

28

Unexpected behaviour from another driver

18

Ford Autonomous Disengagement report

Dec-15

Jan-16

Feb-16

Mar-16

Apr-16

May-16

Jun-16

Jul-16

Aug-16

Sep-16

Oct-16

Nov-16

TOTAL

Miles

590

590

Disengagements

3

3

Per 1,000 miles

-

-

-

5.08

-

-

-

-

-

-

-

-

5.08

For the period Ford had only two identically equipped Ford Fusion Hybrid’s and as mentioned above testing occurred only on Interstate 10 between Los Angeles and Arizona during March 2016. The company announced in December 2016 that it converted 20 Ford Fusion Hybrid Autonomous vehicles for testing in California and Michigan. During the testing, there were only three disengagements, two due to other drivers overtaking at high speed and one due to its self-driving technology malfunctioning.

Honda Disengagement Report

Honda’s report indicates that the company did not test any vehicles on California’s public road, only on a closed circuit, non-public road in Contra Costa County, CA. Honda approached Waymo in December 2016 to explore a partnership similar to Waymo’s successful partnership with Chrysler on the Pacifica Minivan entered into in May 2016.

GM Cruise Autonomous Disengagement report

Dec-15

Jan-16

Feb-16

Mar-16

Apr-16

May-16

Jun-16

Jul-16

Aug-16

Sep-16

Oct-16

Nov-16

TOTAL

Miles

46

61

98

171

256

200

391

570

1,672

2,279

747

2,284

8,776

Disengagements

32

18

7

10

13

14

15

19

20

18

9

6

181

Per 1,000 miles

692.34

295.91

71.18

58.52

50.75

69.99

38.37

33.31

11.96

7.90

12.05

2.63

20.62

GM tested it’s self-driving technology on San Francisco’s city streets using twenty 2017 model Chevrolet Bolt’s, two 2012 Nissan Leaf and three 2016 Nissan Leafs. The program kicked off slowly with only about 2,000 miles done in the Nissan Leafs during the first eight months of the period with around 7,000 miles in the remaining four months done with the new Bolt’s. The bulk of GM’s 181 disengagement events were done to avoid unexpected behavior. Although this resulted in higher incidents than Ford, Nissan or BMW we salute the company for testing in city streets which is ultimately more difficult than testing on interstates and should be taken into consideration when comparing disengagements.

Mercedes Autonomous Disengagement report

Dec-15

Jan-16

Feb-16

Mar-16

Apr-16

May-16

Jun-16

Jul-16

Aug-16

Sep-16

Oct-16

Nov-16

TOTAL

Miles

5

80

68

88

173

259

673

Disengagements

23

125

92

24

27

45

336

Automatic

18

67

48

6

5

9

153

Manual

5

58

44

18

22

36

183

Per 1,000 miles

-

-

4,331.45

1,571.54

1,346.01

-

-

-

-

272.14

156.19

173.64

498.95

Mercedes had only one vehicle doing public testing while the rest of its captured fleet did so on a closed circuit. Mercedes captured automatic disengagements which were technical in nature and manual disengagements where the driver felt uncomfortable. The automaker clearly identified technical issues during the first round of test which they improved in a second round at the end of the year.

Nissan Autonomous Disengagement report

Dec-15

Jan-16

Feb-16

Mar-16

Apr-16

May-16

Jun-16

Jul-16

Aug-16

Sep-16

Oct-16

Nov-16

TOTAL

Miles

1,043

165

20

411

29

85

88

277

58

375

263

1,285

4,099

Disengagements

5

1

-

6

2

2

2

-

-

3

-

8

29

Per 1,000 miles

4.79

6.06

-

14.60

68.97

23.53

22.73

-

-

8.00

-

6.23

7.07

Nissan’s test fleet included three Nissan Leaf’s and two Infinity Q50’s, the automaker’s luxury brand. The company concluded only around 4,000 miles of testing, with over half of it in the first and last month of the test. The automaker claims to have tested on all manner of roads, with 19 incidents on what the classify as Highway and City and ten incidents on what is qualified as City Street and Highway. All of the disengagements were as a result of system failure to identify an incident according to Nissan’s classifications.

Type

Discription

Incidents

1

AV System fails and requires to take over

9

2

AV System does not recognize failure, but the driver needs to take over for safety

0

2A-a

AV is about to collide with another vehicel or obstacle due to self steering

9

2A-b

Another vehicle is about to rear end the AV due to insuficient decelration of the AV

11

2A-c

Another vehicle is about to rear end the AV due to sudden decelration of the AV

0

2B

AV Driver is expecting a failure and can easily handle the disengagement (early development)

0

3

AV driver stops AV Mode voluntarily to finish experiment

0

Tesla Autonomous Disengagement report

Dec-15

Jan-16

Feb-16

Mar-16

Apr-16

May-16

Jun-16

Jul-16

Aug-16

Sep-16

Oct-16

Nov-16

TOTAL

Miles

530

20

550

Disengagements

181

181

Per 1,000 miles

-

-

-

-

-

-

-

-

-

-

341.51

-

329.09

Tesla’s captured fleet for testing on Califonia’s public road system consisted of four Tesla Model X’s. The automaker only tested in October and for a short distance during the month of November. Tesla’s report is quite secretive and not very telling as it does not classify different causes of disengagement other than marking it as “Follower Output Invalid” or “Planner Input Invalid.” Tesla marked a limited amount of entries as “Health Monitor,” “ACC Cancel” or “Cruise Fault.” Over 90% of the disengagements were clocked on wet roads. Tesla’s conducted its tests on freeway’s, suburban and arterial roads. To make any deductions on Tesla based on its disengagement report would be a mistake as the company’s customers have already clocked over 300 million miles in Autopilot, which is operating at Level 2 Automation. Filing as per the DMV requirements is only for Levels 4 and 5 automation. Tesla also divulged in its report that it’s testing in manny other parts of the world.

Discription

Road Condition

Incidents

Planner Output Level

Mostly Wet

76

Follower Output Level

Mostly Wet

99

ACC Cancel

Wet Road

3

Health Monitor

Wet Road

1

Cruise Fault

Dry Road

2

Volkswagen Autonomous Disengagement report

Volkswagen filed report indicate that the company did not do any testing on California‘s public road in the period under consideration.

Waymo Autonomous Disengagement report

Dec-15

Jan-16

Feb-16

Mar-16

Apr-16

May-16

Jun-16

Jul-16

Aug-16

Sep-16

Oct-16

Nov-16

TOTAL

Miles

38,856

38,612

19,870

27,453

38,898

59,489

53,745

74,346

83,705

72,765

72,065

56,334

636,138

Disengagements

2

6

7

7

6

10

9

24

17

12

15

9

124

Per 1,000 miles

0.05

0.16

0.35

0.25

0.15

0.17

0.17

0.32

0.20

0.16

0.21

0.16

0.19

Wyamo has been testing its self-driving technology in four US cities for the past seven years and clocked over 2.3 million miles in Autopilot on public roads. The company improved its disengagement level from 0.8 per 1,000 miles in 2015 to 0.2 in 2016. The company also conducts around 3 million miles a day in simulation mode. Waymo’s captured fleet for testing on California’s public roads consisted of 66 vehicles, and testing was mostly in a suburban city environment in and around Mountain View California. Google breaks its disengagements down further by distinguishing as per the DMV requirements between:

Disengagments

2015

2016

When a failure of the autonomous technology is detected

0.64

0.07

When the safe operation of the vehicle requires that autonomous vehicle test driver disengages the autonomous mode and take immediate manual control of the vehicle

UBER’s self-driving program

Uber was told to suspend its testing on San Francisco road’s by the CA DMV as they could not agree on workable rules around Uber’s demand for its technology to be classified as SAE Level 3 automation. It is not the end of Uber’s efforts though as it is piloting retrofitted VolvoXC90 SUV’s and Ford Focus vehicles in Pittsburgh.