Today we are joined by Autonomous Mobility Strategist and Stantec GenerationAV Founder, Corey Clothier, to discuss Stantec’s approach to measuring and assessing the safety and risk of autonomous vehicles prior to deployment. Corey specializes in helping people solve operational and go-to-market challenges for all things AV, from wheelchairs and personal mobility devices to heavy trucks and delivery robots.
In this episode, we learn about the common challenges companies face when it comes to developing and deploying their AV and how Stantec uses military and aviation best practices to remove human error and ensure safety and reliability in AV operations. Corey explains the importance of collecting edge cases and shares his take on why the autonomous mobility industry is so meaningful.
Key Points From This Episode:
Tweetables:
“For me, [commercialization] is a safe and reliable service that actually can perform the job that it's supposed to.” — @coreyclothier [0:07:04]
“Most of the autonomous vehicles that I've been working with, even since the beginning, most of them are pretty safe.” — @coreyclothier [0:08:01]
“When you start to talk to people from around the world, they absolutely have different attitudes related to autonomy and robotics.” — @coreyclothier [0:09:20]
“What's exciting though is about dRISK [is] it gives us a quantifiable risk measure, something that we can look at as a baseline and then something we can see as we make improvements and do mitigation strategies.” — @coreyclothier [0:17:18]
“The common challenges really are being able to handle all the edge cases in the operating environment that they're going to deploy.” — @coreyclothier [0:20:41]
Links Mentioned in Today’s Episode:
Corey Clothier 0:00
Some of the issues with autonomous vehicle operations aren't the autonomous vehicles. Frequently there's the humans in the loop. So again, that's what we use this digital checklist system as part of our operational safety program to try to remove the human error. And we're just using military and aviation best practices to do that.
Rob Stevenson 0:21
Welcome to how AI happens, a podcast where experts explain their work at the cutting edge of artificial intelligence. You'll hear from AI researchers, data scientists and machine learning engineers as they get technical about the most exciting developments in their field and the challenges they're facing along the way. I'm your host, Rob Stevenson. And we're about to learn how AI happens. Here with me today on the podcast is an automated mobility strategist, global thought leader and innovator focused on commercialization strategies around the future of mobility and automated vehicle systems. He's worked with the Department of Defense, various autonomous vehicle focused companies. He's also the founder of generation AV, which is the autonomous vehicles arm of engineering services firm Stantec Corey clothier, welcome to the podcast. How are you today?
Corey Clothier 1:15
Thanks. I'm doing great, beautiful day in Tampa.
Rob Stevenson 1:18
Glad to hear it. Did I do your curriculum vitae service there? I hope I didn't like you know, skate over anything.
Corey Clothier 1:23
No, it seems that was actually very generous. Saves, like global thought leader. That's a pretty generous statement. But I appreciate it. Thank you.
Rob Stevenson 1:32
Well, yeah, if anyone outside the United States hears this, this technically, you're a global global thought leader. So that's counted. And I have worked.
Speaker 1 1:39
I have worked outside of the United States. I've done projects in Germany, London, and Dubai.
Rob Stevenson 1:46
See you're, you're a man of the world. So Cory, you have so many different projects you're working on at any given moment. I'm kind of curious, just like, what was your week, like this week? Where were you putting your attention on a weekly basis?
Speaker 1 1:59
Oh, my gosh, that's a interesting question. Because I usually do a weekend review on Fridays. And I was I worked with or collaborated with, in some way, an autonomous vehicle company that is doing kind of ride services. With minivans. It's not really a ride share. But that type of service. Also working with a company that's working with autonomous personal mobility devices are essentially the kind of like autonomous wheelchairs, another company that's doing autonomous industrial floor scrubbers, we were we worked on the design for truck ports and terminals for autonomous terminals of the future. And I could probably listed out another three or four, but shows you the diversity. And I really get excited about that. I love how much diversity I have in the world of the ground autonomy.
Rob Stevenson 2:50
That's part of why I was excited to speak with you because when the average consumer, probably not the average listener of this podcast, but when someone hears AV or autonomous vehicle, I think they probably think Tesla, or at least car, but as you just rattled off a couple of examples, all sorts of other vehicles besides the average consumers car. I'm curious about the wheelchair part. So under what circumstance Do you think a wheelchair user would rather be driven around by themselves then use sort of like the a joystick that we are some kind of motorized thing where they're the driver?
Speaker 1 3:20
Yeah. Well, the first applications in the need that we found, I actually heard this probably close to a decade ago, when we were I was working with the department defense, and we were looking for autonomous vehicle applications. And so we were collaborating with all kinds of different folks, but one of them were commercial airports. And I was speaking with a couple of commercial airports, and they were talking about the challenge they had with just the wheelchair porters that it was just so hard to coordinate. And it just seemed like a really great application. But it's taken us a while to actually find the technology providers and get enough focus on it. But an airport is a great example. I have a friend that has a physical disability. He's in crutches like full crutches, he had polio as a child. And he usually uses a wheelchair when he goes to the airport. And he said he sometimes has to wait 30 to 45 minutes before he gets access to a wheelchair. So he has to bake that into his day. And we've actually done some pilots with automated wheelchairs at an airport in Michigan. But that's a great one. But you can see it in any large venue. It could be at sports stadiums, absolutely at hospitals. But we'll likely start doing more trials at airports just because there's demand and the demand keeps going up. It's becoming more and more of a pain point for the airlines in the airports. So hopefully we can solve it.
Rob Stevenson 4:39
That makes sense, I guess. Yeah. It's obvious when you explained like that someone has to bring that wheelchair to the gate for the person who needs the wheelchair. And this happened recently. I remember getting off a plane and their announcement comes over the PA as we're deplaning and the whoever says that we're still waiting on a couple of wheelchairs. I could be a little bit so and then This was an awful faux pas that guy's like, so if you're waiting on a wheelchair just remain seated. So just like an awful turn of phrase there, but yeah, that that was like so those people who needed the wheelchair just like alright, well, I can't leave this plane until someone in this airport, an employee, this airport brings one over to this gate. So yeah, clear use case there. That makes sense. So you spent some time with the Department of Defense working in I think was was it robotics? Is that right?
Speaker 1 5:23
Yeah, ground vehicle robotics group at a research development engineering center in Michigan, they had the mission for all ground vehicle engineering, research and development. And ground vehicle robotics was one of the divisions which we would probably use the term autonomous vehicles, but same thing. Yeah.
Rob Stevenson 5:41
Got it. So what was the state of autonomous at that point? Were there mostly like human operated robot vehicles? Or were they conveying themselves?
Speaker 1 5:49
It was a pretty strong mixture. I think I started working with them in early 2009. And so there were very few fully level four applications that were really commercializable yet, but this was an r&d group. So that's okay. But we were looking at all kinds of different things from squad support vehicles, security, and patrol vehicles. And then a lot of investment was going on in heavy trucks and logistics, and that's still going on today. And the military, there's actually a quote, that's out right now, from the Defense Innovation Unit, there's another heavy truck autonomy, conversion. So in the state, so you asked about the state of autonomy, back in 2009 2010, we weren't level for you the least we were dabbling in level four. But it wasn't really a reliable yet where we could claim that it was really ready for commercial deployment. But that was what we were trying to do is accelerate that commercialization path. And we were doing it through all kinds of different collaboration with industry and other government agencies, even other countries and cities, but it was early days. But you can see the glimmer of hope. It was exciting. And you know, we all had big visions and big optimism for what was coming for autonomy. And we still do, but I think with a little more realism, but over a decade later,
Rob Stevenson 7:05
in that particular case, what is meant by commercialization?
Speaker 1 7:09
That's a good question. Because when you talk about commercialization, I think some of the companies who need the revenue, have a definition of commercialization that may be a little more aggressive than others. For me, it is a safe and reliable service that actually can perform the job that it's supposed to. And reliability is key. But of course, safety is the ultimate factor. So first, I need to truly believe and understand and have been able to test and verify that this vehicle or the system is quantifiably reliably safe, that essentially won't hit anything that won't hit anybody. But then just on the reliability side, for efficiency, it's the same thing, if we're talking about something that I can automate a wheelchair, or maybe we're staying in the airport environment, we're talking about something that is going to move cargo between aircraft or from the aircraft to a cargo center, or kind of an internal distribution center, you just can't have downtime. And that was one of the biggest challenges. And it's still one of the biggest challenges. And those things usually are wrapped around kind of the edge cases, what is confusing the autonomous vehicle, what's kind of stopping in its tracks, I find most of the autonomous vehicles that I've been working with, even since the beginning, most of them are pretty safe. Most of them won't hit anything. But it doesn't mean that they can do the mission.
Rob Stevenson 8:27
Yeah, it's interesting that there's way less forgiveness for downtime with autonomous than there is for human operated vehicle.
Speaker 1 8:35
It's also cultural. So this was kind of interesting. So one of the companies that we piloted was an automated wheelchair company, we piloted at Grand Rapids airport, and it's a Japanese company. And they've been in operation for quite a while, but five or six years, and I think they've been running pilots in at least in one Japanese airport for a couple of years. And I think they said something like they had a dozen and maybe even up to two dozen of these chairs running every day. But when we tried it, it just wasn't reliable enough. Maybe it was, I don't remember the exact numbers, let's just give it a number. Even be generous and say there was 90% reliable, that wasn't good enough. And it was really a, I think, an American attitude. It was the same reliability in Japan. And they were more tolerant, maybe more patient, they really viewed this autonomous robotic technology as a friend and a helper. In the US. It's just the basic tool either works or it doesn't. If it doesn't work, I don't have time for it. It's kind of funny when you start to talk to people from around the world, and they absolutely have different attitudes related to autonomy and robotics. It seems
Rob Stevenson 9:42
like it's empathy, like people have empathy for a human operator who's like, oh, maybe maybe you never know what someone's going through. Maybe they're having a bad day or oh, you know, this is just how it is that we have. It's a robot if it's hardware, if it's metal and plastic and software. It's like okay, this is unacceptable for it to ever not be perfect.
Speaker 1 9:59
Yeah. And we're seeing that with the on road vehicles as well, right? I mean, small crashes make big news. But there's over 40,000 people that die every year in the US alone from traffic accidents. And unfortunately, most of those don't make the news even. It's really sad. So that's a good point.
Rob Stevenson 10:18
Yeah, there was an article this week, I think it was in San Francisco a an autonomous vehicle hit and killed a dog that ran out from behind a car and I read the article. And it seems like it was just a tragedy, but it didn't seem like the car didn't like, go rogue and go into the sidewalk or something. And I shouldn't speak at a school before they've like included what happened. But I was like, Look, dogs get hit by cars a lot by human operators. And it's very sad. But it's no one's like, alright, revoke that person's license, probably. So just the appetite. And the trust, I think is is very different.
Speaker 1 10:47
Yeah, it's kind of funny. Just again, it's back to perception and our tolerance for mistakes, from our tools on our machines. And we should have very, very low tolerance and safety should be paramount and always front of mind. But it reminds me of a project. We did a project that was a one year long pilot in downtown Las Vegas with an automated shuttle. What's a level four shuttle, but it did have an operator on board. And the first couple of hours, there was an accident. And I just left the site. We had all of the hoopla and celebration of launching this vehicle and all of the word rhythm.
Rob Stevenson 11:23
Yeah, yeah, yeah,
Speaker 1 11:25
celebrities given their endorsements, because it was Las Vegas. And I finally left and I was 10 miles away. And I got an email saying your vehicle crash get back here right away. So I go back. And I actually they even sent me a link to the local news and the local news reporters were saying, wow, you know, we just had this crash. We just launched this autonomous vehicle. And it just crashed in the first hours. And boy that just really says a lot about the technology and the readiness, and I know I wouldn't ride on one. Would you ride in one? No, I wouldn't ride in one, and they get back there. And a 18 wheeler truck was downtown Las Vegas train to back up and do an alley illegally. And the shuttle stopped where it was supposed to and the truck backed into it and just barely nicked one of its fenders
Rob Stevenson 12:10
No, come on.
Corey Clothier 12:11
But it was just all blown up out of proportion. It was funny. They did an inspection of the vehicle. They put one of the big novelty band aids on the fender and put it back in service the next day. It was all over autonomous vehicle crash.
Rob Stevenson 12:25
Yeah, exactly. The Rise of the Machines don't trust it. Yeah, exactly. Exactly. The sensationalism. So of course, safety and risk is going to be paramount, with Stantec, who is prepared to call it like a technical engineering services kind of consultancy. Is that right? Yes. So you are visiting lots of different clients, helping them deploy autonomous vehicles and various use cases, when it comes to measuring and assessing safety and risk. What is the approach there?
Speaker 1 12:52
Great question, because it's a new approach. So in why we're doing and obviously, it's couldn't be more important or more critically important to really understand the risks and the hazards and the overall safety of an autonomous vehicle deployment. But there hasn't been ways to measure it until recently. And these were the some of the questions I've been fielding since way back in the military days, when we were launching a pilot at a military base, and the base commander kind of leans over. And the general says, Hey, Cory, how do I know this is safe, and you start scratching your head because you have various levels of certification from the autonomous vehicle supplier, some of them self certify, some of them use third party certification or verification, some don't. And so we started looking at how can we do it as consultants and as the people that are guiding the end user, and the site owners, and even the operators. So I was looking for ways to quantify safety and risk for a couple of years. Then I found this company that was a startup was called de risk. And that was jointly operated out of London and Pasadena, California. And this gentleman love telling the story. So the founder and CEO is His name is Chet Stetson, and he just happens to have a PhD from Caltech in neuroscience and computational programming, I think is you know, that I can usually kind of blend them together and say, computational neuroscience. But you can guess kind of tell where it's thinking is, and he'd worked on various projects with NASA. And then together, they kind of combined efforts. And they created this company called derisk. In their first funding, and their first project was with the UK Government. And it was to create the ultimate driving test for autonomous vehicles. And they do it through an edge case training program. So what they have to do first is collect all the edge cases, I think they have access to a couple 1000 cameras in London, traffic cameras, and then they aggregate all that data and then they sift out the edge cases. It's really neat to see it visually because I think because the way chess thinks in his education, it looks like a brain like a neural network to me and it probably is a neural network, about how he aggregates and organizes the data. So say he believes he has the largest edge case library of any company in the world. And then he can train autonomous vehicles to that. Now, bring in the layer for us, the consulting company that works kind of in the system engineering infrastructure side, deployment side, his what we do now with chest and derisk, as we'll go to the actual odd, and we'll do an odd risk assessment using derisk, just in that odd in that environment, and to identify the specific edge cases that we're going to see within that odd. And it's really, really cool. And the whole simple philosophy is, if we can identify the edge cases that they're going to see within that deployment, and train accordingly. And his trading protocols really cool, it's accelerated, they don't have to do a million scenarios necessarily. Because what he does is picks out the hardest scenarios. And then it's essentially I know, I'm simplifying this and you should probably have chess on your podcast, because it'd be fascinating. He's a much better speaker than I am, obviously, about his technology. But in my simple terms, is that you test for this really, really hard edge case, and you get 10,000 100,000, maybe a million, that if you can solve this, you can solve this other 100,000 problems, and then goes to the next one, and then to the next one, and then creates this really dynamic training scenarios. But that's kind of the overview of it. Again, from a consulting perspective, he's able to show us where the hotspots are, he's able to show us what the edge cases are, some of the mitigation can be done outside of simulation, and outside of the V, essentially, he's like, we've got really poor visibility here, we've got too much traffic here, people don't obey the speed limits, we find all kinds of wild stuff we found in one city, in the downtown area, the residents had a tendency to back out onto the main street, just a weird, local cultural norm for driving behavior is that they backed into traffic, you know, all the time. And so great information for an autonomous vehicle, very to know. But then we'll also see, we found also police on horseback in one city, and nobody was aware of it. But again, the respondent, and all the times we went and visited and did drive throughs, in some type of analysis of the odd and the typical engineering side, just observational for months, none of us knew that they had horses downtown, I could go on, I know we don't have a whole lot of time. But you can tell I start to geek out when you start talking about the specific tools. And what's exciting, though, is about derisk. And gives us a quantifiable risk measure something that we can look at as a baseline. And then suddenly, we can see as we make improvements and do mitigation strategies.
Rob Stevenson 17:44
So cutting edge cases, that is obvious why that's important, because that feels like where the problem was coming in. Right? That's the headline grabby, your AV crashed, because there was this really rare thing that the machine didn't know how to deal with people focus on training the top of the bell curve area, and ignoring these weigh less frequent things, but then one and 100,000 at these scales, you're going to encounter it. But it is curious how solving that edge case then can be applied downstream. This may be as a question for someone who works at D Rose, but I'm so curious how it's applicable to all the other million additional cases.
Rob Stevenson 18:19
Yeah, yeah, I think that it really is fascinating to hear chess, talk about the science behind it. And his AI. How are you? I mean, this is a AI, focused podcast. So you really should talk to chess. It'll be fascinating. He's a great speaker as well. little fun fact about chess as what was that there was a TV show that was like brain games or something like that. It was like brain teasers and phenomenon of the way you can think out your brain, but it was like a weekly television series. And chess was the neuroscientist on it, you know, kind of explaining the brain science behind it. So usually
Rob Stevenson 18:55
is the talking head they went to?
Corey Clothier 18:56
Yeah, so he was he's got some TV time.
Rob Stevenson 19:00
Okay, so he's familiar with the broadcasting thing. Stay tuned audience for hopefully some future episodes with chess. But in the meantime, what is the output then from a company like derisking? Are they just kind of telling your clients like, Hey, here's where it's risky, go fix it, or how prescriptive then are you being to clients to help them iron out the problems are those edge cases,
Speaker 1 19:21
we've kind of just started, we've only done a few of them. But ultimately, what we want to do is, is we want to work with basically all the stakeholders. So if it is like an end user that might be a city or an airport, we want to be able to identify for them where we think the riskiest situations are and what we want to look out for and work on mitigation strategies with them, then we might have downstream from that we may have the operator and that kind of the same thing is work on more physical and field level engineering mitigation strategies. But then ultimately, what we would like to do is have direct work with the autonomous vehicle company and have them trained to those edge cases and simulation Seeing on the test track and then kind of supervised deployment, just ensuring that they can handle all of the edge cases they're going to encounter. So it's kind of a full spectrum approach. And that's our intent.
Rob Stevenson 20:13
So I've had some ad experts on the show who are performing their AV expertise for one company, and you have a unique role and that you get to look into lots of different companies, building autonomous vehicles. So when you look across sort of the the spectrum of companies you're working with, what are the common challenges they're facing when it comes to deploying and developing their AV.
Speaker 1 20:38
Some things are popping into my head, but I have such diversity. And the companies that I work with, I mean, I was even an advisor Honda Indy autonomous challenge. So you know, we're talking race cars, we're talking heavy trucks, delivery robots, personal mobility devices, shuttles, buses. And I'm trying to think the common challenges really are being able to handle all the edge cases in the operating environment that they're going to deploy. But that's definitely one. Another one is, a lot of these companies are fairly early, and their production cycle. So there could be issues with just production quality. At this point, something is loose, or something breaks. So that's where some of the reliability issues we've seen, just because our our industry is so young, we don't have a whole lot of production level autonomous systems yet. And we know who the leaders are out there and how they're doing. But those are common, but what I work with a lot is helping meet people solve more operational challenges and go to market challenges and things like that. The operational side, like we're just so big into operational safety as well. I love talking about the tools that we've curated. Another tool for operational safety is a company called Icarus. It's these pilots. They're all former naval pilots are now they're all airline pilots. But they created a digital checklist system that simply parallels what you see in the commercial aviation industry. But we adapted it and adopted it for autonomous vehicle operation, because what we also find so here's a common challenge, I think is some of the issues with autonomous vehicle operations aren't the autonomous vehicles. I think so frequently, there's the humans in the loop. So again, that's what we use this digital checklist system as part of our operational safety program to try to remove the human error. And we're just using military and aviation best practices to do that.
Rob Stevenson 22:38
Yes. Well, you mentioned military and aviation and best practices. A friend of mine was the helicopter mechanic and Marines for about 20 years. And I asked him, helicopters are like very dangerous, right? Don't they crash out? He's like, Yeah, it's almost always human error. And that was surprising to me. When they do all these investigations of why these things crash and it's never like the hardware failed, right? It was a human decision. That feels like the true north of AVS. Like why AVS? Like it's not to economize away labor from people it feels like driving a car, for example, is the most dangerous thing any of us do? Probably on an on a daily basis for you. Is that the opportunity behind AV? What do you think is so appealing about this industry? Besides that, it's, it's cool that robots can drive themselves.
Speaker 1 23:19
Yeah, really a safety first. I mean, that was what really inspired me and was inspiring. Everybody that I was working with at department defense, is that one of the highest causes of death in the US military, even during wartime sometimes was vehicle accidents. So we were really trying to solve that. And then of course, during combat operations, vehicles are targets. And her intent was to try to save lives. And then I geek out further on just the operational efficiencies. And then I have these dreams of being able to not be in traffic jams someday, I don't know if that'll happen in my lifetime. But the vision of that of having a fully automated driving, whether surface streets and highways, where we can sit back and the robots into computers can manage the traffic, so it flows all the time. What a dream, and I really, truly believe that it's going to happen someday. I have no idea if it's gonna happen in 20 years or 120 years, but I really think that's our future. And if I can play a small part in that gets me very excited.
Rob Stevenson 24:24
As a Once and Future loss Angeleno the idea of solving traffic really gets me excited. So Cory, I wish you the best of luck. I hope you are the one to shepherd for that paradise. Before I let you go though, Cory. I just want to have you share one more thing for the folks out there forging a career in autonomous vehicles, what advice would you give them to make sure that they're forging an impactful career?
Speaker 1 24:43
Hmm, that's a big question. Because I have three adult children, one that I work with full time she's her and I've been working in the autonomous space for years together, and then two that I bring in from time to time. So they all are very familiar with the space But they're super unique in their talents and their career paths. And I was just talking to my daughter yesterday just about my career and my life because you mentioned the Marine Corps helicopter mechanic. I started in the Marine Corps as a jet avionics technician, and then went to flight school and did that route then I was in automotive engineering, and I was in innovation and blah, blah, blah, we were just talking about how all of our careers are so different. And just, I feel really lucky that I was always able to do something interesting. I always gravitated towards interesting and impactful and much I really have appreciated my career. So that's, that's essentially and I've probably talked to two or three young people that are starting their career a month, and they're always interested in autonomous vehicles. And it's really just finding what they're passionate about, match their passion with their skill set, and what makes them smile every day. So it's just waking up and being excited about what you're going to do every day is the most important thing.
Rob Stevenson 25:57
That's fantastic advice, Cory. I have had such a blast chatting. This has been fascinating. Thank you so much for being here and for sharing your background and expertise with us. I've loved learning from you today.
Corey Clothier 26:07
Appreciate it, there's been a lot of fun.
Rob Stevenson 26:11
How AI happens is brought to you by sama. Sama provides accurate data for ambitious AI specializing in image video and sensor data annotation and validation for machine learning algorithms in industries such as transportation, retail, e commerce, media, med tech, robotics and agriculture. For more information, head to sama.com