Even if neural nets continue to come up with adequate solutions to challenging problems (sometimes, hopefully, maybe), at the end of the day the best-case scenario is a model that cannot be audited, holistically understood, or trusted with decisions of any gravity.
If beating the strongest human Go players with a self-trained ML model was detonating a fission bomb, pervasive automation with zero accountability is a nuclear winter.
You can be cynical and say humans are utterly unreliable like AI model reasoning. But it's not really true.
I regularly go to the grocery store and purchase things produced by human from around the country and around the globe. In the process of producing things and getting to the grocery isles, I'm sure there are always fuck-up on various people's parts and there are other people who compensate for them. That behavior is far what AI can do at present.
That isn't saying AI isn't impressive in many ways. But it's a brute-force simulation of what people do and it's fragility demonstrates this.
A human can be held responsible for a mistake. A model can't. For the ethically flexible, the latter is a benefit rather than a hazard: one can use ML to whitewash bias and rely upon public perception of computers as dispassionate and objective.
Humans are expensive and require individual training. Models scale. In the time it would take a human to make a mistake, a broadly-deployed model might make millions of impactful mistakes.
I think analogy with medicine is fruitful. Small molecules can't be held responsible either, but they can be recalled from market and they do. Our pharmacology isn't good enough to design, say, vaccine that works worse for Asians, but we did require clinical trial participants to be a representative sample. Medicine is also mass produced, that's why there is post-market surveillance to catch rare side effects missed in clinical trials.
Even with all this analogy, we don't insist medicine should have known mechanism of action, although we do prefer it. So I think we will regulate models with recalls, testing standards, monitoring etc, but won't insist on understanding.
By the way, don't we already have widely deployed models, such as PhotoDNA, which supposedly removes millions of images a year to filter child pornography? I wonder how it was evaluated to be suitable for deployment.
We are fine with approving medicine with no known mechanism of action but with safety and efficacy shown in clinical trials. We will be fine with models.
Medicine isn't making any decisions, AI or humans are. We, as the humans, need to understand and assess why the AI is making a decision and is it the right decision.
If beating the strongest human Go players with a self-trained ML model was detonating a fission bomb, pervasive automation with zero accountability is a nuclear winter.