A California bill could seriously limit open-weight AI models
How SB 1047 could discourage Meta from publishing big open-weight models in the future.
I’m a journalist with a computer science master’s degree. In the past I’ve written for the Washington Post, Ars Technica, and other publications.
Understanding AI explores how AI works and how it’s changing the world. If you like this article, please sign up to have future articles sent straight to your inbox.
SAN FRANCISCO—California state Sen. Scott Wiener faced a skeptical crowd Thursday at the headquarters of Y Combinator, the legendary startup accelerator. Wiener is the author of SB 1047, legislation that would require creators of very large AI models to test them and certify that they are unlikely to cause major harms.
Critics worry this heightened liability could discourage companies like Meta from releasing open-weight models in the future. That would be a particular blow to startups, many of which use open-weight models because they cannot afford to train their own models from scratch.
“I support open source,” Wiener said. “I think open source brings so many benefits to the open source ecosystem. My goal is not to shut down open source.”
Yet critics argued that the legislation would do exactly that—at least for the largest AI models. The next speaker was Andrew Ng, a machine learning pioneer and critic of Wiener’s legislation.
“When I look at some of the requirements, I would have a hard time knowing what exactly I would need to do to comply with them,” Ng said at Thursday’s event. “This is one of those moments where such laws, not just in California but around the world, could have a huge impact on whether entrepreneurs and startups are allowed to keep on innovating.”
Wiener’s bill could become law in a matter of weeks. It passed the state Senate in May with a lopsided 32 to 1 vote. Insiders tell me that the bill has a good chance of passing the state Assembly. Opponents’ best chance to stop it may be a veto by Gov. Gavin Newsom.
I’ve spent the last week talking to both supporters and critics of the legislation. It’s a confusing debate because the bill has evolved during the legislative process. Wiener has listened to critics and narrowed his legislation in important ways. Some criticisms focus on provisions that are no longer in the bill.
But the critics are still correct on a fundamental point: the bill really could discourage companies from releasing large open-weight models in the future. And this isn’t surprising if you understand where the bill came from.
The bill’s most important co-sponsor is the Center for AI Safety Action Fund, a group headed by computer scientist Dan Hendrycks. Hendrycks believes that there is an 80 percent chance that powerful AI systems will pose an existential risk to humanity in the coming years. If large models were really that dangerous, then of course they require strict regulation and shouldn’t be made freely available for anyone to use or modify.
But if large models don’t pose existential risks, then SB 1047 seems like overkill. Because open-weight models are valuable for startups, academic researchers, and anyone who wants to run large language models on their own devices.
The case for open-weight models
On Tuesday, Meta announced Llama 3.1, a new “herd” of large language models. The release included Meta’s long-awaited model with 405 billion parameters. In performance terms, the new model seems to fall a bit short of leading models from Google, OpenAI, and Anthropic. But it was by far the most powerful open-weight model released up to that point. The Llama weights can be freely downloaded, used, and modified by anyone.
Why is Meta giving away models that cost millions of dollars to train? Meta CEO Mark Zuckerberg explained his thinking in an essay published alongside the new models.
“Meta has a long history of open source projects and successes,” Zuckerberg wrote. “We’ve saved billions of dollars by releasing our server, network, and data center designs with Open Compute Project and having supply chains standardize on our designs. We benefited from the ecosystem’s innovations by open sourcing leading tools like PyTorch, React, and many more tools.”
Zuckerberg argued that open-weight models would be good for the world too.
“Open source will ensure that more people around the world have access to the benefits and opportunities of AI,” he wrote, “that power isn’t concentrated in the hands of a small number of companies, and that the technology can be deployed more evenly and safely across society.”
Lina Khan, the chair of the Federal Trade Commission, made a similar argument at Thursday’s Y Combinator event.
“With open-weight models, more small players can bring their ideas to market,” she said. “There is tremendous potential for open-weight models to promote competition across the AI stack and by extension, spur innovation across the stack, too.”
SB 1047 wouldn’t apply to the Llama models Meta released this week—not even the model with 405 billion parameters. Meta says that training this model required 3.8x1025 floating point operations (FLOPs). That’s less than half the 1026 FLOP threshold that triggers potential liability under SB 1047.
But given the exponential growth of large language models in recent years, it’s easy to imagine Meta’s next generation of language models exceeding the 1026 FLOP limit. If SB 1047 is the law in California at that point, Meta could face a new set of legal requirements.
SB 1047 would require Meta to beef up its cybersecurity to prevent unauthorized access to the model during the training process. Meta would have to develop the capacity to “promptly enact a full shutdown” of any copies of the model it controls. Most important, Meta would have to write a safety and security policy that “provides reasonable assurance” that the model will not pose “an unreasonable risk of causing or enabling a critical harm.”
Under the bill, “critical harms” include “the creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties,” “mass casualties or at least $500 million of damage resulting from cyberattacks on critical infrastructure,” and “mass casualties or at least $500 million of damage” from a model that “acts with limited human oversight, intervention, or supervision.” It also covers “other grave harms to public safety and security that are of comparable severity.”
A company that violates these requirements can be sued by California’s attorney general. Penalties include fines up to 10 percent of the cost of training the model as well as punitive damages.
Crucially, these rules don’t just apply to the original model, they also apply to any derivative models created by fine tuning. And research has shown that fine tuning can easily remove safety guardrails from large language models.
That provision about derivative models could keep Meta’s lawyers up at night. Like other frontier AI developers, Meta has trained its Llama models to refuse requests to assist with cyberattacks, scams, bomb-making, and other harms. But Meta probably can’t stop someone else from downloading one of its models and fine-tuning it to disable these restrictions.
And yet SB 1047 could require Meta to certify that derivative versions of its models will not pose “an unreasonable risk of causing or enabling a critical harm.” The only way to comply might be to not release an open-weight model in the first place.
Some supporters argue that this is how the bill ought to work. “If SB 1047 stops them, that's a sign that they should have been stopped,” said Zvi Mowshowitz, the author of a popular Substack newsletter about AI. And certainly this logic makes sense if we’re talking about truly existential risks. But the argument seems more dubious if we’re talking about garden-variety risks.
General-purpose technologies have both good and bad uses
People do a lot of useful things with trucks, but it’s also possible to turn a truck into a bomb by filling it with fertilizer and diesel fuel. If we required truck manufacturers to certify that their trucks would not be used for critical harms, it might be impossible for them to comply. Yet it would be crazy to effectively ban trucks because a few people might turn them into bombs.
By the same token, it seems very plausible that people will use future large language models to carry out cyberattacks. One of these attacks might cause more than $500 million in damage, qualifying as a “critical harm” under SB 1047.
But those same models are likely to also be useful for auditing systems to defend against cyberattacks. Indeed, the defensive uses of LLMs might vastly outweigh their offensive uses, making the Internet more secure overall.
It’s not clear whether companies are allowed to consider tradeoffs like this under SB 1047. If a Llama model gets used in a major cyberattack and Meta gets sued, could the company argue that the risk was reasonable in light of the substantial offsetting benefits of its model? It’s hard to predict whether that would stand up in court.
And the problem is that big companies and their lawyers are a conservative bunch. If a law has multiple interpretations and some of those interpretations involve millions of dollars of liability, a good lawyer is going to advise their client to play it safe. For Meta, that would mean not releasing open-weight models that require more computing power than the thresholds set by SB 1047.
Wiener has significantly narrowed SB 1047
The version of SB 1047 passed by the Senate in May applied to any model trained with 1026 FLOPs or any model that “could reasonably be expected to have similar or greater performance” as a model trained on 1026 FLOPs in 2024. Given the continued operation of Moore’s Law and rapid improvements in AI algorithms, this language would have meant more and more models coming under the law’s purview over time.
The Senate bill also allowed California officials to change the compute threshold, raising fears that overzealous regulators might lower it in the future.
Facing heavy criticism over these provisions, Wiener made a big concession as the bill moved to the state Assembly: he removed the language about “similar or greater performance” and added a new requirement that the value of a model’s training compute be more than $100 million.
This change won’t have a big impact in the short run, since 1026 FLOPs of computing power costs roughly $100 million today. But as Moore’s Law continues to make computing power cheaper, the $100 million requirement will allow companies to train more and more powerful models without triggering SB 1047. And crucially, the $100 million figure can’t be changed by regulators.
It’s possible that this new provision will render SB 1047 toothless. A big trend in recent months has been frontier labs releasing mid-sized models that outperform the previous generation’s largest models. In other words, these companies are learning to use their compute budgets much more efficiently. And of course Moore’s Law makes FLOPs cheaper every year.
If these trends continue, it’s possible that AI companies will never want to spend more than $100 million to train a frontier model. In that case, they’ll never need to worry about the requirements of SB 1047.
Certainly that would be a welcome outcome for fans of open-weight models. But I think it would count as a failure for those who believe powerful models pose an existential threat to humanity. Whatever dangers AI models pose is a consequence of their capabilities, not their compute budget. A law that exempts ever more powerful models is nothing to celebrate if you’re worried about powerful AI models taking over the world.
This may be a compromise many AI doomers are willing to make because they don’t expect the cost of powerful models to plateau any time soon. Many agree with Leopold Aschenbrenner’s projection that companies will be spending hundreds of billions of dollars on training clusters by the end of the decade. And it’s possible that the cost of training frontier models will get so high that no one—not even Mark Zuckerberg—can afford to give away the model weights.
But I think there’s a good chance we’ll end up with a scenario somewhere between these extremes: where training costs rise above $100 million but don’t rise so quickly that Meta gets priced out of its open-weight strategy. And in this scenario, SB 1047 could dramatically alter the structure of the AI market, with a proprietary tier of higher-performing models and an open-weight tier of much weaker models. It’s little wonder that fans of open-weight models are organizing to make sure that’s not the future we get.
Like so many California politicians, these Ai safety saviors want to gain power and control the rest of us in order to aggrandize their own careers and satisfy their own kneejerk submission to fear. California is shooting itself in the foot with this legislation. Vote against this bill and anyone who supports it.
It says in one of the image captions that you interviewed Lina Khan. Is there any place we can view/read that?