'Doomsday cults' and the Silicon Valley debate over AI’s threat to humanity

By VCU News staff

The conversation about artificial intelligence took on new shape recently after artificial intelligence researcher Jacob Coxon publicly resigned from Anthropic, citing concerns about long-term safety.

Following Coxon's resignation - and the subsequent headlines it generated - a group of tech leaders, including the CEOs of Anthropic and OpenAI, raised the alarm on their own models, highlighting the dangers they could pose.

While there are critics quantifying the AI risk in terms of possible human extinction, others decry these warnings as fearmongering, with some experts labeling those sounding the alarm as a doomsday cult.

On paper, this looks like a disagreement. But this messaging actually stems from well-funded and connected Silicon Valley communities known as rationalists and effective altruists. Caddie Alford, who has been researching Big Tech and the effects of algorithmic systems since 2014, said that these groups have had enormous influence on the proliferation of AI, with Coxon's own warnings reflecting similar ideology.

At Virginia Commonwealth University, Alford, Ph.D., is an associate professor of rhetoric and writing in the Department of English in the College of Humanities and Sciences whose research looks at tech giants and their influence on culture, information, sociality and democratic civic discourse. She co-leads the interdisciplinary AI Futures Hub, which pursues critical humanities approaches to AI, Big Tech platforms and the effects they have on labor, sociality, culture, the environment and democratic discourse.

Alford is also collaborating on an upcoming book about how rhetoric can help diagnose and rectify technofascism. VCU News caught up with her for insight into the state of the AI conversation.

First, give us some background on a key player in this debate and the overall AI landscape?

Eliezer Yudkowsky, who dropped out of school after eighth grade, founded the nonprofit Singularity Institute for Artificial Intelligence in 2000. The goal of SIAI was to "create a friendly, self-improving artificial intelligence." So for over 25 years, Yudkowsky has promoted the idea that AGI - artificial general intelligence - or superintelligence is possible.

At first, he was all-in on trying to make an all-powerful AI. And as a kind of feeder team for SIAI and as a side project, he created the LessWrong blog in 2009 to help people expand so-called "rational" approaches to their cognitive biases.

Rationalism is not the only Silicon Valley ideology pursuing AGI. Computer scientist Timnit Gebru and philosopher Émile P. Torres created a term - the TESCREAL bundle - to name them, including Transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism and Longtermism - some ideologies that figures such as Peter Thiel and Facebook co-founder Dustin Moskovitz have self-espoused.

What's notable about these ideologies is that they all extend eugenics-based premises and arguments. For example, some TESCREALists argue that AI will optimize humans into a superior species of posthumans, with extended "longevity." Singularitarianism is the belief that algorithms will self-improve past human capacity, or what is known as "the singularity." Longtermism promotes directing resources toward the future, at the expense of present problems like food scarcity and disease prevention.

And how did this "doomsday cult" notion evolve?

Today, the SIAI is called Machine Intelligence Research Institute, and MIRI's mission appears to reflect a 180-degree turn: "Do research and public outreach intended to help prevent human extinction from the development of artificial superintelligence." The MIRI homepage features the cover of Yudkowsky's co-authored book: "If Anyone Builds it, Everyone Dies: Why Superhuman AI Would Kill Us All."

The LessWrong community of rationalists has pivoted, too, in Yudkowsky's direction. They are now embedded in Silicon Valley culture, with active groups all around the world. And yes, they are now what is broadly billed a doomsday cult.

And do they have a large footprint in the AI industry?

This all seems super-niche, but rationalists have had an outsized influence on both Silicon Valley and the advancement of AI technologies and company power.

LessWrong is widely read by OpenAI and Anthropic researchers and engineers, and it has been for years. Yudkowsky is a primary reason why investor Peter Thiel financially supported OpenAI, and Dario and Daniela Amodei, who left OpenAI to co-found Anthropic, have ties to TESCREAL communities.

In 2023, Thiel warned OpenAI CEO Sam Altman at a dinner party that Yudkowsky "has programmed half the people in your company to believe" in the doomer outlook.

To me, someone who studies persuasion and the intersection of power and language, the choice of the word "programmed" tells me he knows - that they all know - that what they are doing, and how easily they can get it done without the public catching up, is contingent on rhetoric.

Give us a practical impact of this.

Yudkowsky and rationalism helped make AGI seem inevitable. OpenAI's mission since 2015 has been to build AGI, and the unchecked race to do so is directly responsible for large language models and platforms integrating GenAI features - all to buy time to figure out how to make all this profitable and how to keep feeding the beast (as in, keep users locked into platforms handing over text and data).

In no uncertain terms, the goal of creating AGI - a goal that rationalists are directly responsible for - resulted in the recent hype-driven AI summer, not even four years since the November 2022 release of OpenAI's ChatGPT.

So, where do things stand as we forge ahead in an AI world?

Rationalists and other TESCREAL groups believe that AI is superpowerful and will develop superintelligent capacities that, unless they are the ones in control, will want to wipe out humans. Others see a business opportunity to create more capital, and some see this as a classic ploy for AI companies to generate buzz about their products while ensuring regulatory capture. These frequently are all the same people.

All of these beliefs - and all of this power - have been forming across a long history that can't be separated from the inception of cybernetics in the 1940s, the 1956 Dartmouth AI Summit, the creation of the chatbot Eliza in the '60s and the history of the internet.

Because of all that money and power, it's also tough because we are talking about groups that spend a lot of time talking just to other believers who also have beliefs that amount to religiosity and science fiction. What someone like Yudkowsky says publicly does not map onto what he really wants. Yudkowsky may be called a "doomer," but he is all-in on AI.

We're supposed to think the accelerationists, who want rapid transformation through privatization and advancement of AI products, and the doomers are different. But they both want control over these technologies, they both share ideological commitments, there's a lot of money on the table - and ultimately, they both seem to want these technologies to become whatever they mean by "superintelligences."