Firing Sam Altman hasn't worked out for OpenAI's board

If you come at the king, you’d best not miss.

Sam Altman. (Photo by Justin Sullivan/Getty Images)

On Friday, the board of OpenAI fired CEO Sam Altman, kicking off a weekend of frantic negotiations. As I write this on Monday, Altman has been hired by Microsoft, and more than 500 of OpenAI’s employees have threatened to follow him if OpenAI’s board doesn’t meet their demands.

The board’s actions are inexplicable if you think of OpenAI as a traditional tech startup. But OpenAI isn’t a tech startup—it’s a non-profit organization with a complex history, an unusual corporate structure, and a mission to “to ensure artificial general intelligence benefits all humanity.” It was co-founded in 2015 by Altman and Elon Musk, both of whom expressed grave concerns about the dangers of artificial intelligence technology.

“When Musk and Altman unveiled OpenAI, they also painted the project as a way to neutralize the threat of a malicious artificial super-intelligence,” Cade Metz wrote in a 2016 Wired profile of the OpenAI. “Of course, that super-intelligence could arise out of the tech OpenAI creates, but they insist that any threat would be mitigated because the technology would be usable by everyone.”

Nick Bostrom, the philosopher whose 2014 book inspired Musk and Altman’s concerns, didn’t find this convincing at all. “If you have a button that could do bad things to the world, you don’t want to give it to everyone,” Bostrom told Wired in 2016.

Tensions over this question have dogged the organization ever since. They ratcheted up in recent years as OpenAI raised money to build and commercialize more capable models. And they finally boiled over with Friday’s decision to fire Altman.

The board hasn’t explained Altman’s firing in any detail. But independent reporting suggests that the board had grown uncomfortable with Altman’s aggressive timeline for commercializing GPT-4 and other OpenAI technologies. OpenAI’s recent developer day event, which focused on helping people integrate OpenAI’s chatbot technology into a wide range of apps, probably didn’t assuage concerns that OpenAI was moving too quickly.

The problem facing the board now is that there’s no easy way for OpenAI to return to its roots as an elite research organization. If OpenAI isn’t going to aggressively commercialize its technology, it can’t afford to employ hundreds of top-tier AI researchers or spend billions training new models. OpenAI’s top talent is likely to follow Altman to Microsoft, leaving OpenAI as a shell of its former self.

And ultimately, this isn’t going to do much to slow the growth of potentially dangerous AI. It’s just going to shift power over AI to companies like Microsoft and Google that may be even less concerned about safety than OpenAI was under Altman.

The origins of OpenAI

Today OpenAI is synonymous with ChatGPT and other generative AI systems. But transformer-based language models didn’t exist yet when OpenAI was founded in 2015. In the early years, OpenAI’s research largely focused on reinforcement learning, a technique that was particularly well-suited for playing video games and controlling robots.

After Google invented the transformer in 2017, OpenAI started experimenting with the technology. OpenAI released GPT-1 in June 2018 and announced GPT-2 in early February 2019. 

OpenAI usually made its research freely available in the early years, but it made an exception for GPT-2. GPT-2 was far less capable than today’s frontiers models, but OpenAI was still concerned about it “being used to generate deceptive, biased, or abusive language at scale.”

Around this time, OpenAI researchers noticed what became known as “scaling laws”—the fact that larger language models do a better job of predicting the next word in a passage. OpenAI wanted to press forward with GPT-3, making it much bigger than GPT-2. But a larger model would require computing power to train, and it was difficult for a traditional nonprofit to raise the necessary funds. So OpenAI created a for-profit subsidiary and had it raise $1 billion from Microsoft.

Crucially, the non-profit parent organization retained full control. OpenAI made clear to investors that it would prioritize its mission over the interests of investors. On its website, OpenAI states that “investors could lose their capital contribution and not see any return.” It adds that “it would be wise to view any investment in the spirit of a donation.”

So on paper, Microsoft had effectively given OpenAI $1 billion to do as it saw fit. In practice, it wasn’t that simple. As OpenAI hired more staff and trained larger models, it became dependent on further fundraising to continue operating. That gave Microsoft practical leverage over OpenAI on an ongoing basis.

OpenAI released GPT-3 in 2020 and GPT-3.5 in early 2022. Then near the end of 2022, OpenAI released ChatGPT, a version of GPT-3.5 that was optimized for conversation. ChatGPT was a bigger hit than OpenAI leaders expected, and its success started to change OpenAI’s corporate culture.

I recommend yesterday’s article in the Atlantic explaining how the success of ChatGPT heightened internal tensions at OpenAI. The company shifted computing resources away from research groups as it scrambled to keep up with surging demand for ChatGPT. The process strengthened the hands of OpenAI employees who favored faster commercialization, while increasing resentment among safety-minded employees.

“As Altman and OpenAI President Greg Brockman encouraged more commercialization, the company’s chief scientist, Ilya Sutskever, grew more concerned about whether OpenAI was upholding the governing nonprofit’s mission to create beneficial AGI,” Karen Hao and Charlie Warzel write in the Atlantic. “Over the past few years, the rapid progress of OpenAI’s large language models had made Sutskever more confident that AGI would arrive soon and thus more focused on preventing its possible dangers.”

Crucially, Sutzkever held a seat on OpenAI’s board. Indeed, due to attrition, the board had only six members—Altman, Brockman, Sutzkever, and three independent directors. Those three ultimately sided with Sutzkever against Altman and Brockman.

The power of effective altruists

Sutzkever wasn’t the first OpenAI employee to grow disillusioned with OpenAI’s rapid growth. In 2021, a group of OpenAI researchers left to start a competing organization, Anthropic. The Anthropic founders have never clearly explained their reasoning, but it’s widely believed that they thought OpenAI was taking inadequate safety precautions.

One of the Anthropic founders was married to OpenAI board member Holden Karnofksy. Karnofsky was an executive at Open Philanthropy, a foundation devoted to effective altruism that gave at least $30 million to OpenAI in its early years.

After Karnofsky resigned his board seat to avoid a conflict of interest, it went to Helen Toner, a former Open Philanthropy staffer who is now at Georgetown’s Center for Security and Emerging Technology. CSET was created with a $55 million grant from Open Philanthropy.

Another OpenAI board member is Tasha McCauley, a tech entrepreneur who serves on the board of Effective Ventures, a charity that incubates nonprofits committed to effective altruism. Open Philanthropy gave Effective Ventures more than $11 million in 2022.

I’m not suggesting that there was anything inappropriate about these relationships. But I think it’s hard to understand the events of the last few days without noting the outsized influence that Open Philanthropy—and the effective altruism movement more broadly—plays in contemporary AI policy.

When I was working on my story on Joe Biden’s executive order, multiple DC insiders told me CSET scholars had a big influence on the White House’s AI policy thinking.

Last month Politico published an expose explaining how people with ties to Open Philanthropy have gained senior positions in AI policy roles across the government. Funded by Open Philanthropy, a group called the Horizon Institute for Public Service was paying the salaries of AI policy staffers in several Capitol Hill offices. Other Horizon alums were working in policy roles at several federal agencies.

The Politico piece portrayed this an example of powerful Silicon Valley companies promoting their self-interest, but I think that actually misrepresents what was going on. Open Philanthropy’s main funder, billionaire Facebook cofounder Dustin Moskovitz, wasn’t funding Horizon fellowships to get a better return on his investments in OpenAI and Anthropic. It was actually the opposite: he invested in OpenAI and Anthropic because he is sincerely worried that a rogue superintelligent AI might take over the world. He’s an ideologue, not a rent-seeker.

So when four OpenAI board members—Sutzkever, Toner, McCauley, and Quora CEO Adam D'Angelo—voted to oust Altman, I assume they believed they were protecting humanity from future attacks by superhuman AI.

The board’s problem was that it lacked the support of rank and file workers at OpenAI. On Monday, more than 500 OpenAI employees—out of around 700 total—signed an extraordinary letter threatening to follow Altman to Microsoft if the remaining OpenAI board members didn’t resign en masse.

So it seems unlikely that Altman’s ouster will lead to more cautious development of AI technology. Whether Altman ultimately stays at Microsoft or comes back to OpenAI, he’ll be more powerful than he was last week. And if he wants to rapidly develop and commercialize powerful AI models, nobody will be in a position to stop him.

Remarkably, one of the 500 employees who signed Monday’s OpenAI employee letter is Ilya Sutskever, who has had a profound change of heart since he voted to oust Altman on Friday.

“I deeply regret my participation in the board's actions,” he wrote on X this morning. “I never intended to harm OpenAI. I love everything we've built together and I will do everything I can to reunite the company.”