That is seen, by AI companies such as Google-owned Deepmind, as one of the possible triggers for a scenario in which AI vastly exceeds human intelligence (known as artificial superintelligence) instead of just matching it (known as artificial general intelligence).
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon said in a follow-up post.
Evan Hubinger, Anthropic’s staff lead on keeping the technology aligned with human goals and values, backed up Coxon claims in a follow-up post of his own, though he didn’t quit the company.
“Jacob is correct here — we really do earnestly believe AI could kill all humans,” he said.
Hubinger estimated the chances of that happening to be higher than ten percent within the next decade, and added that there’s no plan yet on how to keep AI aligned with human goals in the superintelligence scenario.
Last week, U.S. Senator Bernie Sanders announced he would introduce legislation to ban firms from developing superintelligence.