An autonomous car can have zero failures and still be unsafe
Bright sunlight on a camera lens breaks nothing, yet it can make an autonomous vehicle do something it was never meant to do. That gap, between a system that has not failed and one that is truly safe, is where much of the remaining work lies. Senior Safety Engineer Palak Talwar of Lyft's self-driving division walks through it, and the harder question behind it: how would anyone know a self-driving car is safe enough?
Learn the technical and regulatory hurdles standing between autonomous vehicles and public roads
Palak Talwar, Senior Safety Engineer at Lyft's self-driving division, gives a beginner-friendly overview of autonomous vehicle safety, from autonomy levels and vehicle architecture to the technical and regulatory hurdles that remain. She argues that functional safety (ISO 26262) is only a starting point: SOTIF, verification and validation, cybersecurity, liability and the lack of an agreed safety metric all stand between AVs and deployment at scale.
“Now, when you talk about functional safety, functional safety is only focused on detecting system failures and reacting to those failures, right?”
“But how do you know it's safe enough? So I think defining a metric, or, you know, how do you measure safety is a big question or a challenge, you know, for AVs.”
What you will learn
Distinguish functional safety (ISO 26262), which handles system failures, from SOTIF, which handles unintended behaviour when nothing has failed.
Map the four pillars of an autonomous vehicle: sensors, autonomy software, vehicle controls and electrical systems.
Explain the levels of autonomy from L1 driver assist to geofenced L4 and full L5.
Recognise why defining a metric for safe enough is the central open problem in AV verification and validation.
Identify the regulatory hurdles to deployment: proving customer safety, the shift of liability to manufacturers and AV-ready infrastructure.
Key takeaways from the speaker
Autonomous Vehicle Safety
Details
Level
Beginner
Format
Talk
Technical depth
Introductory technical
For
Student, Early career, Mid-career
Length
17:28
Published
June 2021
Questions this talk answers
What is the difference between functional safety and SOTIF?
Functional safety, under ISO 26262, focuses on detecting system failures and reacting to them. SOTIF, safety of the intended functionality, covers cases where nothing has failed but sensor limitations or inadequate training data make the vehicle behave in unintended ways.
Why is it so hard to prove an autonomous vehicle is safe enough?
Teams can run many hours of testing on the bench, in the field and on the vehicle, but there is no agreed metric for safety. Organisations such as IEEE, SAE and AVSC are working on how to define one.
Who is liable when an autonomous vehicle causes an accident?
Current liability laws already make manufacturers liable for safety incidents. As driving responsibility shifts from the driver to the car, liability may shift from the driver to the manufacturer too.
How soon will autonomous vehicles be commercialised at scale?
Talwar does not expect it right away or within the next couple of years, since L4 autonomy is still limited to geofenced areas and regulatory standards are not yet in place. Going from L4 to L5 will be a big jump.
How will AI affect autonomous vehicle safety?
AI and machine learning are interlinked with perception, prediction, localization and planning in the autonomy stack. Making sure neural networks have adequate training data and that perception errors are minimised is where the big challenge lies.
About this talk
The research and development of autonomous vehicles can potentially revolutionize the transportation sector. However, as several countries worldwide prepare to roll out the technology, that will take over driving functions from testing stages to commercial phases, safety measures and regulations remain a significant concern. In addition, the anticipated shift in driving responsibilities from a driver to an autonomous vehicle serves as a motivation for researching and analyzing liability and safety concerns. Failing to develop or observe safety regulations and standards exposes autonomous vehicle users to high risks, including unexpected accidents that may cause death. More importantly, reasonable and precise determination of safety liabilities are essential to encourage manufacturers to research further how to ensure autonomous vehicles are safe and secure.
SOTIF: when nothing fails but the car is wrong (54 s)
Why AVs must prove customer safety first (65 s)
How do you know it's safe enough? (64 s)
So, hello everyone. I hope you're all keeping safe. My name is Palak and I work as a senior safety engineer at Lyft Level 5, and I also act as a safety advisor for the University of Toronto self-driving car team. I'm really excited to speak at the conference today and share some thoughts around autonomous vehicle safety challenges. And with that, let's get on to the first slide.
Now, before we get into talking about some key technical and regulatory challenges for AVs, I'd like to take a moment to talk about the level of transportation that has increased over the past years, from early cars with minimal electronics and with driver performing all the functions, to L1 autonomy with driving assist features like ACC, adaptive cruise control, or lane keep assist, to L4 autonomy, where the driver is not required to intervene at all. However, for L4 autonomy, there are some restrictions as full autonomy mode can only be activated in certain geofenced areas, all the way to L5 autonomy, where the vehicle is capable of performing all functions under all conditions through ability to sense its surroundings. Now, as the level of transportation has increased over
the past years, the level of safety has also increased significantly, all the way going from having minimal electronics to support the driver, to having multiple levels or layers of autonomy software to the driver's stack with vehicle, you know, enabling all the functions or vehicle performing all the functions under all conditions through ability to sense its surroundings. Now, before I get into, you know, to talk about some of the key technical and regulatory challenges when it comes to deploying AVs on public roads and at scale, I'd like to give a bird's eye view of different pillars of autonomous vehicles so that, you know, when we get into challenges, it'll basically give you a good understanding of the scope of those challenges.
So, starting with sensors, now starting with at the very top of the vehicle, we have sensors, sensors, you know, different sensors like camera, radar and LIDAR, they collect or gather information 360 degrees around the vehicle. That information is fed into autonomy. Now, sophisticated algorithms, you know, they gather or they process the sensor data, enabling the autonomous vehicles to understand its environment. algorithms like neural networks, they help process information, learn and classify the objects in the surrounding environment. That information is fed into vehicles controls, that is engines, engine brakes and steering system to operate the vehicle or to control the vehicle.
Last but not least, we have electrical. Electrical signals that moves through the vehicle's controls that connect the computer to the engine brakes and steering system to operate the vehicle. Now, these different pillars of autonomous vehicles, they comprise of key technical terms that you might have used or even heard before. And some of them are just highlighted here, like perception, prediction, planning and localization that fall under autonomy software stack that help process the information, they process the sensor data. And then based on where the predicted or the motion of, you know, the surrounding objects is going to be, it basically plans a route for the vehicle. And that information is fed to vehicles controls. And along with that, we have some other terms like power management,
weather proofing and integrated design. Now, each pillar in here, it has its own complexity. And when you combine these different pillars of autonomous vehicles, the level of safety just increases altogether. And, you know, with that, let's talk about some of the key technical challenges for AVs. Starting with managing complexity at scale. Now, traditional automotive companies, for the past so many years, they have been manufacturing vehicles with compliance to functional safety standard, that is ISO 26262. Now, with autonomy stack and with added complexity, the level of safety increases significantly. All the way going from or ranging from detecting sensor failures, like detecting camera, radar or light or sensor failure, to effectively detect, localize and classify objects in the surrounding environment is one of the challenges when it comes to minimizing perception errors. Also, to make sure
that the vehicles controls, like brakes, steering and engine, they operate or they basically send out correct and in a timely fashion actuator commands is also another area where safety plays an important role. Now, when you talk about functional safety, functional safety is only focused on detecting system failures and reacting to those failures, right? So what happens if there is a sense of failure? What happens, you know, if the system or the autonomy software stack does not, you know, detect the obstacle correctly? What happens if there is a delay in the actuator command to the vehicle? So all of these are categorized under system failures. Now, consider a scenario where there is no system failure, but because of some technological limitations like sensor performance limitation or inability to address or inability to
comprehend a situation and operate safely or inadequate or having inadequate training data set to, you know, operate under different or in different for different conditions or under different conditions. Now, because of these, you know, corner cases or use cases, which is outside of system failure, that the output of the function is unintended. Or in other words, the vehicle is not doing what it's supposed to do. And this is where the second challenge or even the second standard comes in the picture, which is called SOTIF. Now, SOTIF stands for safety of the intended functionality. SOTIF standard focuses on, you know, providing design, verification and validation measures. And applying these measures, it just, you know, helps achieve safety without failures. So if there is no system failure, then how do we make sure that the system or the functionality of the vehicle is still
intended, that it's supposed to, it's doing what it's supposed to do. Now, you know, someone might ask this question, like, how do we achieve safety? In other words, how do we measure safety? How do we know that the system is safe enough, you know? Or in other words, how do you define a metric for safety, for the safety of the system? And this is where the third challenge comes in the picture, verification and validation. Now, with again, going back to the previous thing with autonomy stack and the added complexity, the level of safety increases. And another hand, in parallel, the level of testing that's needed, it also increases significantly, you know. For autonomy stack, you have to make sure that you have, you know, captured all the use cases and corner cases that does not just rely on system failures,
but also on these sort of specific use cases. For example, if there is a bright sunlight on camera lens, right? It's still doing what it's supposed to do. There is no system failure or malfunctioning behavior, but it might not work the way it's supposed to, which could lead to some unintended functionality. So making sure that, you know, define a metric for that, it's a big challenge.
Last but not least, we have cybersecurity. Now, you know, due to the extreme connectivity within autonomous vehicle components with other vehicles, that is V2X, or in other words, vehicle to vehicle communication and their operating environment, one of the challenges is to protect fleet and customers from cybersecurity attacks. You know, connectivity additions include like IT backend system or new interfaces between the connected functions or between the control functions of the connected vehicles and other external information sources. This rich cyber attack or this rich attack surface, it basically creates considerable interest for malicious actors with lots of, you know, with, again, going back to the complexity and going back to integrated design of the system.
Going to some of the regulatory challenges, you know, along with these key technical challenges, where it's still, you know, a lot of things are unknown and we are still working on defining standards for those. There are also some regulatory challenges when it comes to deploying AVs on public roads and at scale. Starting with proving customer safety, and this is something that, you know, a lot of people that I've worked with or even, you know, I talk to, they, you know, answer, basically think about this.
And even I do sometimes, and I started to work on safety. That is proving customer safety right now. Before any reasonable on-road rollout, customer or consumer safety has to be paramount with rigorous testing, with rigorous testing for all autonomous vehicle components. Now, if you think of it in terms of statistics, in a recent survey, two fifth respondents, they expressed comfort with low-speed AVs. However, that dropped when asked about a high-speed environment.
In other words, or another way to think of it is, the expectation from AVs is to reduce the number of crashes or especially fatal crashes as compared to human-driven cars, right? Because they are run by machines that have less scope of errors. And if AVs are not able to establish this clearly, it'll be a significant setback for the adoption and rollout of AVs. The second one that you think about is liability, right? Now, current liability laws, they mandate manufacturers to be liable for safety incidents.
Now, because of the steady shift in vehicles' responsibility from the driver to an autonomous car, may see a similar change in liability from the driver to the manufacturer. So, developing unsafe products implies that manufacturers will be legally responsible for accidents involving their products. Last but not least, we have building AV conducive infrastructure, which is where, you know, government will have to closely monitor the development and evolution of AVs. And they'll have to determine the impact this technology will have on road infrastructure, our cities and communities, when it comes to deploying AVs on public roads and at scale.
Short and sweet from my end. And I'm just next to Q&A, as I'm open to, you know, answer any questions that you might have. Or I'm happy to take any comments, suggestions from any of you. And obviously, you know, happy to connect to you on LinkedIn. Thank you so much, Falak. And now we're seeing a wild kaleidoscope of your shared screen. There you go.
Yeah, I mean, so far from your conversation, there's a few things I think that stand out. I know sometimes when you're kind of moving quickly to to get all the information in for us, there's sometimes things that you think to yourself, I would have spent more time there. But I wanted to make sure, you know, everything was said correctly within my time. Seeing as we do have a little bit more time, is there one piece you'd love to kind of like give us a little bit more on while we have you?
Sure. I think when it comes to, you know, implementing safety strategies or key different principles of safety, right? When you talk about building safety principles for AVs, we basically started with functional safety as a basis. As I mentioned before, you know, traditional automotive companies, they have been manufacturing vehicles with compliance to this ISO 26262 or functional safety standard for like so many years. When we talk about implementing safety principles for AVs, we basically start from scratch. In other words, we start from the basic. So we have to apply functional safety, which, you know, helps us define some key safety management related principles, right? Beyond is where, you know, a lot of unknowns and a lot of these corner cases and use cases as it comes in the picture. And which is where I think a lot of focus is going to go in and a lot
of standards or a lot of organizations, either IEEE or SIE, they have been working on right now, even AVSC, which is a safety consortium. They have been trying to, you know, get to a point where you, you know, how do you achieve safety? How do you define a metric to safety, you know, for AVs? Because it's really difficult to, when someone asks me the question, like, you know, how do you know whether it's safe enough? You can do X number of hours of testing, you can, you know, you can do different levels of testing, you can test it, you know, at different levels, you can test it on the vehicle, you can go to a field to test the vehicle, or you can do it on the on the bench, right, which is like a specific
test setup for the safety. But how do you know it's safe enough? So I think defining a metric, or, you know, how do you measure safety is a big question or a challenge, you know, for AVs. Do you feel like, Palak, like the speed of these safety testing and mechanisms is going at the pace that you expect it should go?
Oh, I guess, yes or no. I mean, all the tech companies they're working on right now, they are doing, you know, amazing job, right? But I guess it comes back to the question of what are they, like, whatever they are doing, is that good enough? Again, going back to the same thing, right? The speed at which they're going, I don't think that commercializing AVs is going to happen right away, or even for the next couple of years, it's going to take a while for us to commercialize AVs, right? And as you're talking about L4 autonomy, right now, we are only able to engage in autonomy for most of the, you know, tech companies that you talk about. We're only able to engage in autonomy and test it out just in certain areas or geofenced areas, right?
But commercializing that over like everything, it's going to be a big challenge. And I don't think we have the regulatory standards in place yet to even get to that. So even if we go to that L4 for autonomy, going from L4 to L5 is going to be a big jump. And yeah. Do you think regulatory standards will happen in some countries before others, like where legalization will happen earlier in certain areas of the world? And if so, is the US one of them?
Where's the kind of fastest moving process on the regulation side? Um, that's more on the regulatory side. I'm not so sure I know a lot more about regulatory, because it goes back to compliance and legal side of things. But I know a lot of folks are actually working on it and Europe being one of those. I'm not sure if they are, and as compared to US, whether they are, you know, ahead or they are behind. But, but yeah.
Yeah. Well, I asked from that country just because that's where you're the office that you're at is located. But I think so interesting. So like, where, what do you see the future as in this realm? Um, just sit back and relax, take a nap, you know, read a book, and try not think about safety, you know, and this is where the regulatory challenge, it's very important, because proving customer safety. And, you know, we'll have to make sure that there are different ways to do that. But I guess we'll have to make sure that we prove customer or consumer safety, deploy, or start to deploy AVs on public roads, and even at scale. Because if we can do that, then I think the only thing that I'm going to do is just
sit back, relax, take a nap, or enjoy a drink. And, you know, just let the car do its job. Yeah, yeah, absolutely. And I'm sure people are commenting and thinking that same thing. That'd be the best we could get to work on my commute and not need to do anything. Well, I have a question in the chat, too, for you. How do you think AI is going to impact on autonomous vehicles specifically?
Well, I guess a lot of it, when you talk about autonomy, software stack, right, AI or machine learning, I think a lot of it is just interlinked together. So it's, you know, it's not interdependent. They're also dependent. And when you think about all of these different levels of autonomy, software stack, like prediction, perception, localization, or planning, this is actually the bigger piece of us to ensure safety, right? We could control is something that we have been doing in doing it, you know, before as well. And for us to make sure that we are able to send out correct actuator commands to the vehicle, we still have some idea of how to do that.
But making sure that they're getting correct or adequate training data set from neural networks, or we are getting or we don't have our minimizing perception errors that we're able to detect obstacles correctly, and in a timely fashion, right? How do you make sure of that? I think this is where it's actually a big challenge. And this is where I think AI and machine learning is going to come in the picture, of course.
Thank you.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why an autonomous vehicle can have no failures and still be unsafe
Lyft safety engineer Palak Talwar explains why functional safety is only the starting point for self-driving cars, and what still blocks deployment at scale.
In short
Autonomous vehicles cannot be made safe simply by detecting and handling component failures. Palak Talwar, Senior Safety Engineer at Lyft's self-driving division, argues that the hardest problems sit beyond traditional functional safety: unintended behaviour when nothing has broken, the lack of an agreed metric for "safe enough", a growing cyber attack surface, and regulatory questions about customer safety, liability and infrastructure.
Key takeaways
Distinguish functional safety (ISO 26262), which handles system failures, from SOTIF, which handles unintended behaviour when nothing has failed.
Map the four pillars of an autonomous vehicle: sensors, autonomy software, vehicle controls and electrical systems.
Explain the levels of autonomy from L1 driver assist to geofenced L4 and full L5.
Recognise why defining a metric for safe enough is the central open problem in AV verification and validation.
Identify the regulatory hurdles to deployment: proving customer safety, the shift of liability to manufacturers and AV-ready infrastructure.
From driver assist to full autonomy
Talwar, who also advises the University of Toronto self-driving car team, begins with how far vehicles have come. Early cars had minimal electronics and the driver did everything. L1 autonomy added driving assist features such as adaptive cruise control and lane keep assist. At L4 the driver does not need to intervene at all, though full autonomy can only be activated in certain geofenced areas. L5 is the end point, where the vehicle performs all functions under all conditions by sensing its surroundings.
To frame the challenges, she gives a bird's eye view of four pillars. Sensors such as camera, radar and lidar gather information 360 degrees around the vehicle. Autonomy software, built on algorithms such as neural networks, handles perception, prediction, planning and localization. Vehicle controls (engine, brakes and steering) act on those decisions, and electrical systems carry the signals between the computer and the controls. Add power management, weatherproofing and integrated design, and the safety task grows quickly.
"Now, each pillar in here, it has its own complexity." (3:38)
Functional safety is only half the story
Traditional carmakers have built vehicles in compliance with the functional safety standard ISO 26262 for years. Talwar explains that with an autonomy stack the safety scope widens, from detecting sensor failures to correctly detecting, localizing and classifying objects, and to making sure brakes, steering and engine receive correct actuator commands on time.
"Now, when you talk about functional safety, functional safety is only focused on detecting system failures and reacting to those failures, right?" (4:39)
The harder cases are the ones where nothing fails. Sensor performance limits, an inability to comprehend a situation, or an inadequate training data set can all produce output the system was never meant to produce. That is the territory of a second standard.
"Now, SOTIF stands for safety of the intended functionality." (5:35)
SOTIF provides design, verification and validation measures to achieve safety without failures. Talwar's example is simple: bright sunlight on a camera lens. "For example, if there is a bright sunlight on camera lens, right? It's still doing what it's supposed to do." (7:16) No component has malfunctioned, yet the vehicle may not behave as intended.
How safe is safe enough?
As complexity grows, so does the testing burden. Verification and validation must capture use cases and corner cases that go well beyond system failures, and Talwar names the underlying problem directly: nobody yet has a settled way to measure safety. In the Q&A she expands on this. Teams can test for hours, on the bench, in the field or on the vehicle, and bodies such as IEEE, SAE and the AVSC safety consortium are working on the question, but the answer remains open.
"But how do you know it's safe enough? So I think defining a metric, or, you know, how do you measure safety is a big question or a challenge, you know, for AVs." (13:19)
Cybersecurity is the fourth technical challenge. Vehicle to vehicle (V2X) communication, IT back ends and new interfaces between connected functions and external information sources create a rich attack surface that, she says, attracts malicious actors. Protecting the fleet and customers becomes part of the safety case.
Customer safety, liability and the road ahead
On the regulatory side, Talwar starts with proving customer safety, which has to be paramount before any reasonable on-road rollout. As the speaker cited, in a recent survey two fifths of respondents were comfortable with low-speed AVs, but that comfort dropped for high-speed environments. The public expects AVs to cause fewer crashes, especially fatal ones, than human drivers, and failing to show that would be a significant setback for adoption.
Liability is the second issue. Current laws already make manufacturers liable for safety incidents, and as driving responsibility moves from the driver to the car, liability may move too. "So, developing unsafe products implies that manufacturers will be legally responsible for accidents involving their products." (10:04) The third issue is infrastructure: governments will need to monitor AV development and judge its impact on roads, cities and communities.
Asked whether progress is moving fast enough, Talwar praises the work underway but does not expect commercialisation at scale right away or within the next couple of years. L4 testing remains confined to geofenced areas and regulatory standards are not yet in place.
"So even if we go to that L4 for autonomy, going from L4 to L5 is going to be a big jump." (14:28)
She closes on AI: machine learning is woven through perception, prediction, localization and planning, and ensuring adequate training data and minimal perception errors is where she sees the biggest challenge, and where AI will matter most.
Palak Talwar is a Senior Safety Engineer at Lyft, Self Driving Division, after about four years at an automotive consulting company working on control system software development and functional safety. She holds an approved patent on control redundancy and an ISO 26262 certification.
Questions this talk answers
What is the difference between functional safety and SOTIF?
Why is it so hard to prove an autonomous vehicle is safe enough?
Who is liable when an autonomous vehicle causes an accident?
How soon will autonomous vehicles be commercialised at scale?
How will AI affect autonomous vehicle safety?
Drafted with AI from the talk transcript. Editor review pending.