What you're describing is a driving test that every human needs to go through before they're allowed to drive on public roads. Something that can be revoked temporarily or permanently.
I would be very interested in a 3rd party (government or private) create a rigorous test (obstacles, weather conditions, etc...) for self driving vehicles. Becoming "XXX safety Certified with a Y Score" for all major updates to the AI could help restore confidence in the system and eliminate bad actors.
In some countries you have to pass a medical examination which includes a vision test.
The driving schools also have theoretical tests where one has to identify objects in a picture, interpret the situation, and propose the correct action. Of course, these tests are on a higher level: "this is a car, this is a pedestrian" vs. "you're approaching an intersection, that car is turning left, a pedestrian is about to cross the street, another car is coming from there etc."
Not to mention the road and track tests a driver has to pass which include practicing controlling the car in difficult conditions: driving in the dark, evasive actions on slippery surfaces and so on.
Edit: In my opinion it's insane to allow autonomous vechiles on the roads without proper testing by a neutral third party.
I would be very interested in a 3rd party (government or private) create a rigorous test (obstacles, weather conditions, etc...) for self driving vehicles. Becoming "XXX safety Certified with a Y Score" for all major updates to the AI could help restore confidence in the system and eliminate bad actors.