Governor Newsom vetoed California's most controversial AI safety bill
AI regulation should focus on systems, not models.
The most high-profile AI policy fight of 2024 ended yesterday when California Governor Gavin Newsom vetoed SB 1047, Sen. Scott Wiener’s legislation to regulate frontier models.
“By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology,” Newsom wrote in his veto message.
He added that the bill “does not take into account whether an Al system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions—so long as a large system deploys it.”
As I’ve followed the debate over SB 1047 in recent months, I’ve noticed that supporters had a range of arguments about why it was needed.
Some worried that AI models would achieve superhuman intelligence, develop agendas of their own, and disempower or kill all human beings. Notably, this seems to have been the principal concern for Dan Hendrycks, who as head of the Center for AI Safety was an official sponsor of Wiener’s legislation. He has stated that his P(Doom) is 80 percent, meaning that he is very concerned about existential risk from AI.
But perhaps anticipating that this concern would seem far-fetched to many policymakers, Wiener did not focus on it when advocating for his legislation. More often, he talked about relatively pedestrian risks such as hackers using AI for cyberattacks. Wiener frequently described SB 1047 as a “light touch, commonsense measure.”
I think this approach might have worked if Wiener had focused solely on proprietary models. Big AI labs like OpenAI, Google, and Anthropic were already taking many of the precautions mandated by SB 1047, and they could have easily afforded to comply with the law’s other provisions.
Wiener’s conundrum was what to do about open weight models. A company like Meta can add guardrails to a model like Llama. But once the weights are public, there’s no way to prevent someone else from fine-tuning the model to strip off the guardrails. So if you want all large AI models to comply with a set of safety rules, you basically can’t allow large open-weight models.
SB 1047 didn’t ban large open-weight models outright, but it came pretty close to it. The law would have made the original creator of an open-weight model responsible for derivative models created by others. If Newsom had signed the bill, it’s easy to imagine Meta’s lawyers deciding it was too risky to release open-weight models above the size threshold set by SB 1047.
The potential loss of large open-weight models was a major motivating factor for technologists who organized against SB 1047. And it seems to have influenced Newsom’s thinking. A few days before his veto, Newsom said he was concerned about the “chilling effect, particularly in the open source community, that legislation could have.”
Regulate systems, not models
In my view Wiener made a mistake by focusing on AI models rather than AI systems. An analogy might help to illustrate the difference.
Suppose it’s 1984 and policymakers are beginning to worry about harmful uses of computer chips. They predict, for example, that in the future terrorists could use a computer chip to control and detonate a bomb remotely. They worry that creeps could use computer chips to violate other people’s privacy—for example by hiding tiny digital cameras in locker rooms. Eventually, computer chips might enable novel nuisances like spam and hacking.
So a far-seeing legislator introduces legislation requiring computer chip makers to institute a thorough testing regime. Before shipping a new chip, companies like Intel and Motorola would need to certify that it had taken reasonable care to avoid producing a chip that (in the words of SB 1047) “poses an unreasonable risk of causing or materially enabling a critical harm.”
The problem with this is that safety is not a property of computer chips. Chips are flexible, general-purpose technologies that can be used for a variety of purposes—some helpful and others harmful. Whether a chip is being used in a harmful or dangerous way depends on the design of the overall device: for example, the same image sensor might be used in a regular smartphone or a spy camera designed for peeping toms.
The same principle applies to AI models. Nobody ships a bare AI model to consumers. Rather, someone has to combine the model with other components to build a finished product like a chatbot, a medical diagnostic tool, or a self-driving car.
To regulate an AI system sensibly, you need to understand both the system architecture and the broader context. Is the system capable of taking action in the physical world? To whom is the product being marketed? Is there human oversight of the model’s decisions?
A single general-purpose model can be used in many different systems; it’s not reasonable to expect the creator of a model to anticipate and prevent all harmful uses.
I view this as orthogonal to questions about “how much” to regulate AI. SB 1047 really was a light touch bill in some ways. It gave AI companies a lot of leeway to decide for themselves what harms their testing should focus on.
Conversely, AI regulations that focus on systems rather than models could easily wind up being stricter and more burdensome to companies shipping AI products. That’s precisely the worry of Dean Ball, a libertarian-minded critic of SB 1047.
I share some of Ball’s concerns here. I just think we’re going to have much more fruitful debates if it’s clear what we’re arguing about. Should we have stricter regulation of self-driving cars? Do we need to streamline the FDA approval process for AI-powered medical apps? Should it be legal to sell apps that generate fake nude images of real people? Are companies using AI hiring systems that discriminate against racial minorities?
Asking focused questions like this seems far more promising than asking whether an AI model is dangerous or not—a question that’s simply too vague to produce a meaningful answer.
Conversely, the risk of regulating finished systems, rather than models, is that we might be too late: released open-weight models are, as you note, not that hard to modify, replicate, and hide. Even closed-weight models can be leaked or stolen. If they're powerful enough to have dangerous abilities, the public gets screwed. Without regulation, tech companies have insufficient incentive to prevent this.
If automated intelligence will only ever be as dangerous as computer chips, then only regulating systems will be fine. If it might be as dangerous as nuclear weapons, then I suppose we'd want to regulate its components more like fissile material.
Sam Altman was on the latter side [checks notes] sixteen months ago. How time flies.
typo: should be Wiener, not Weiner