Driverless cars that are not actually reliable can be easily fooled.

Judged as of 2019-09-20. True: published research or documented incidents from up to that date show that driverless car perception systems then in use or under development could be misled by simple or cheap manipulations (e.g. stickers on stop signs, small road markings, projected images, adversarial patches) or by unusual situations. Mostly true: such vulnerabilities were shown, but mainly in lab or research settings rather than on deployed commercial vehicles, while still being easy to produce. Mixed: vulnerabilities were shown but required sophisticated or costly attacks, so 'easily' is only partly supported. Mostly false: the only evidence was narrow, single-component attacks that did not carry over to full driving systems, and the claim generalises from them. False: the evidence shows that 2019 driverless systems were robust to such manipulations.

Set before the research began.

1. E1 Tencent Keen Security Lab: Experimental Security Research of Tesla Autopilot Tencent Keen Security Lab · 29 March 2019

   “Based on the research, we proved that by placing interference stickers on the road, the Autopilot system will capture these information and make an abnormal judgement, which causes the vehicle to enter into the reverse lane.”

2. E2 Tencent Keen Security Lab: Experimental Security Research of Tesla Autopilot Tencent Keen Security Lab · 29 March 2019

   “This is not a real-world concern given that a driver can easily override Autopilot at any time by using the steering wheel or brakes and should be prepared to do so at all times.”

3. E3 Physical Adversarial Examples for Object Detectors National Science Foundation Public Access Repository (hosting 12th USENIX Workshop on Offensive Technologies, WOOT 18) · 4 August 2018

   “With a perturbation in the form of only black and white stickers, we attack a real stop sign, causing targeted misclassification in 100% of the images obtained in lab settings, and in 84.8% of the captured video frames obtained on a moving vehicle (field test) for the target classifier.”

4. E4 Researchers trick Tesla Autopilot into steering into oncoming traffic Ars Technica · 2 April 2019

   “Researchers have devised a simple attack that might cause a Tesla to automatically steer into oncoming traffic under certain conditions.”

Before 2019-09-20, independent research showed that driving perception systems could be made to give wrong outputs by cheap manipulations such as stickers on stop signs or on the road [E1][E3]. Peer-reviewed work used black-and-white stickers on a real stop sign to cause targeted misclassification in 84.8% of video frames captured from a moving vehicle (E3). Tencent Keen Lab showed that stickers on the road made a production Tesla's Autopilot misjudge the lane and steer into the reverse lane (E1), and Ars Technica called this a simple attack (E4). The Keen Lab report also includes a statement saying the attack was not a real-world concern because a driver can override Autopilot (E2), but that does not change the finding that the system made an abnormal judgement, which is what the 'true' criterion asks for.