We ask almost every AI architect and leader the same thing in private: What AI risk worries you most?Almost all of them fire back the same response: a killer pathogen, spreading too silently, widely and quickly to stop.Why it matters: The rare agreement among AI experts flows from the belief these advanced models could help a bad actor create a deadly pathogen before the government and industry perfect detection and prevention systems.This isn't a likely scenario. And some of AI's biggest optimists reject doom scenarios.It's simply a plausible one as AI gets better at understanding our biological vulnerabilities, much like it has our cyber ones inside large nonhuman systems.Driving the news: The experts aren't hedging. In a new MIT FutureTech and University of Queensland study, 272 researchers ranked 24 top AI risks.They assessed a 12% chance that AI's dangerous capabilities produce a catastrophic outcome by 2030, and another 12% chance of AI-enabled weapons and mass-harm capabilities.And that's with mitigation efforts to reduce risk.
Without those efforts, the chances exceed 20%.The researchers defined "catastrophic" as more than 1 million deaths or $100 billion in damage. Assisting with the construction of chemical or biological weapons sits squarely inside that top-ranked risk category.The big picture: We're not writing this to scare you, but to say bluntly what those building the technologies say privately — and, in more careful ways, publicly.Just last month, OpenAI's Sam Altman, Anthropic's Dario Amodei, Google DeepMind's Demis Hassabis, Microsoft's Mustafa Suleyman and Meta's Alexandr Wang cosigned an open letter warning of the risk of AI-derived bioweapons and calling for more safeguards.This pathogen possibility colors every debate by the federal government and industry about how to review and understand new AI models and capabilities before they are released to the public.The hope is that the frontier model developers build guardrails and capabilities that can block bad actors with bad intent — and that the very technology that could be used to create a pathogen becomes advanced enough to prevent or inoculate against it.Reality check: It's very hard to prevent bad actors from plotting evil schemes, especially with so-called open-source models improving so rapidly.These open models can be used and adapted by anyone in private settings outside any r…
Summary from source