When Failure Is Not an Option: Neil Siegel on AI’s Dangerous Blind Spot

Neil Siegel has spent his career building systems where mistakes cost lives. Ambulance dispatch networks, air traffic control, the Minuteman missile system - 450 nuclear warheads, 60 years, zero accidental launches. So when he says AI is not ready to run the world's critical infrastructure, he's not speculating. He's pattern-matching against decades of hard-won engineering judgment.

"The algorithms used in these AI engines are based on mathematics that cannot - absolutely cannot - bound either the frequency or the magnitude of the errors they make," said Siegel, IBM Professor of Engineering Management and Professor of Engineering Practice at the USC Viterbi School of Engineering, in an interview filmed for The BlackVeil Files. "You ask 10 questions, and nine times it makes a pretty reasonable answer. And the tenth is wildly wrong. And if you depend on the tenth, you would die."

The full interview, available on YouTube, cuts to the core question animating the AI debate: not whether these systems are impressive, but whether they are trustworthy enough for the infrastructure society depends on most.

Two Worlds

Siegel draws a sharp line between what he calls the "toy business" - Silicon Valley's dominant culture, built around speed-to-market and features - and the world of mission-critical systems where he built his career. The distinction isn't a slight; it's structural.

"If Facebook goes down for a while, it's an annoyance. If an ambulance dispatch system goes down, people die." The companies now building AI engines, he argues, grew up in the first world - optimized for consumer delight, not societal reliability. "They send out a patch to their software every 17 seconds. In our business, we don't do that. Because if you patch an airplane, you have to redo the airworthiness certificate. That costs $100 million."

The mismatch becomes dangerous, he warns, not because Google or OpenAI will deploy their engines in a hospital - but because someone else will. "My worry isn't that the people who build these AI engines will put them in an ambulance dispatch system. My worry is that somebody else will. And they probably already are."

The AGI Question

Asked about artificial general intelligence - the idea that AI will eventually surpass human cognition across the board - Siegel is characteristically blunt: "I have a lecture I give my students where I go through the mathematics and physics of why I don't think that's possible."

His argument draws on Kurt Gödel's incompleteness theorems, Alan Turing's proof that some problems cannot be solved by any algorithm, and Roger Penrose's theory that the human brain operates at a quantum scale that is inherently non-deterministic. "Whatever the brain is," Siegel says, "it's not a computer. It does not work that way."

He acknowledges he's in the minority. But the implications matter: if the brain can't be replicated algorithmically, then AI - however sophisticated - will keep guessing when it doesn't know. "The AI engines are financially incentivized to guess and not tell you they're guessing. If you had a multiple-choice test where getting it wrong cost you five points, you'd leave it blank. We build society's critical systems more like that: no guessing, and being explicit about when we are unsure. Psychology teaches us that humans make better decisions when they are aware of the uncertainties involved, than when they are hidden."

The Regulation Imperative

What would responsible AI deployment look like? Siegel's answer is unambiguous: a regulatory body with teeth, modeled on the FDA or the FAA. Specifications, failure thresholds, testing protocols - the same infrastructure that governs drugs and aircraft. The regulators, not the builders, would decide when something is good enough.

"You can't even sensibly talk about an AI infrastructure without it. It would be like saying we're going to allow a prescription drug business with no qualification procedure." He pauses. "Two hundred years ago, anyone could be a physician and anyone could make and sell drugs. We decided that wasn't good enough."

The AI industry, he predicts, will fight regulation hard. But the scaffolding will come - the question is how much gets built before something serious goes wrong.

Watch the full interview below.

About Neil Siegel

Neil G. Siegel is the IBM Professor of Engineering Management and Professor of Engineering Practice with Distinction in the Daniel J. Epstein Department of Industrial and Systems Engineering at the USC Viterbi School of Engineering, where he also holds a joint appointment in the Thomas Lord Department of Computer Science within USC Viterbi and the USC Stevens School of Computing and Artificial Intelligence. A triple USC alumnus - earning his B.S., M.S. and Ph.D. from the university - Siegel has built one of the most decorated careers in American engineering.

In 2023, President Joe Biden awarded him the National Medal of Technology and Innovation, the nation's highest honor for technological achievement. He is an elected member of the National Academy of Engineering (2005), a Fellow of the National Academy of Inventors (2020), an IEEE Fellow and recipient of the IEEE Simon Ramo Medal (2011), and a Fellow of INCOSE (2021), the International Council on Systems Engineering.

Before joining the Viterbi faculty, Siegel retired from Northrop Grumman as Sector Vice President and Chief Technology Officer, where he oversaw more than 12,000 scientists and engineers and an annual R&D portfolio of approximately $600 million. He led the development of landmark defense and intelligence systems, including the Army's Blue Force Tracker and pioneering unmanned aerial vehicle programs. He holds nearly 50 issued and pending patents spanning defense technology, wireless networks, healthcare and systems engineering, and served on both the U.S. Defense Science Board and the U.S. Army Science Board.

Siegel is also the son of Judith Love Cohen, a USC alumna and aerospace engineer whose work on the Apollo Lunar Module's abort-guidance system was instrumental in the rescue of Apollo 13.

Published on November 10th, 2026

Last updated on November 10th, 2026