California should rethink its broad and sloppily drafted deepfake bill

AB 3211 could hamper open weight models and mandate annoying notifications.

Dean W. Ball is an AI policy analyst at the Mercatus Center and the author of the Substack newsletter Hyperdimensional.

---

There’s been a lot of attention paid to SB 1047, the AI safety bill from California state Sen. Scott Wiener. As Tim wrote on Friday, SB 1047 could discourage companies from releasing large open-weight models, putting small technology companies at a disadvantage. The bill has other flaws that I have written about here and here.

But SB 1047 is far from the only AI bill worth discussing. It’s not even the only one of the dozens of AI bills in California worth discussing. Let’s talk about AB 3211, the California Provenance, Authenticity, and Watermarking Standards Act, written by Assemblymember Buffy Wicks, who represents the East Bay.

Assemblymember Buffy Wicks. (Photo from the California Assembly)

Like SB 1047, AB 3211 has passed one chamber of the California legislature, so it has a real chance of becoming law. And it would have a big impact on the AI industry—perhaps even bigger than SB 1047.

One big impact could be to seriously hamper the development of open weight generative models. The bill bans any website from hosting AI systems that don’t include a watermarking feature. Depending on how you read the bill’s somewhat confusing definitions, that could force HuggingFace to take down most of the generative models on its site.

The bill also requires every generative AI system to maintain a database with digital fingerprints for “any piece of potentially deceptive content” it produces. This would be a significant burden for the creator of any AI system. And it seems flatly impossible for the creators of open weight models to comply.

Under AB 3211, a chatbot would have to notify the user that it is a chatbot at the start of every conversation. The user would have to acknowledge this before the conversation could begin. In other words, AB 3211 could create the AI version of those annoying cookie notifications you get every time you visit a European website.

Deceptive deepfakes are a legitimate problem, and legislation may be needed to deal with it. But this bill is sloppily drafted and goes far beyond the steps taken by deepfake bills in other states. It might be best for Assemblymember Wicks to shelve her bill for this session and come back next year with narrower and more carefully crafted legislation.

The big picture

The bill centers on standards for watermarking synthetic content. A “watermark” can mean many different things, both in the context of this bill and in the broader AI world. It might refer to metadata attached to an AI-generated video or image that specifies the model that produced it, the date it was made, etc. In the context of text-based models, it often refers to having the model output subtle patterns that can be detected by a simple algorithm; to you, it just looks like a normal chatbot response, but to the right algorithm, it is a hidden message saying, in essence, “this was written by AI.”

The problem is that, as of writing, these watermarking standards don’t work very well. I’ve written before about the flaws of C2PA, which is a technical standard for attaching provenance metadata to images and videos. The main issue with C2PA is that it is trivial to remove or change the metadata it attaches.

One could remove the metadata directly, by stripping it from the file, or indirectly, by, for example, taking a screenshot of the image. The screenshot you take will be identical to the AI-generated image, yet will have none of the C2PA metadata conveying that the image was generated by AI.

With text, it’s even easier, because all you need to do is, well, edit the text. As the user, you don’t know which words in the model’s output are being used to encode the watermark, so it would be easy to inadvertently remove.

AB 3211 mandates “maximally indelible watermarks,” which it defines as “a watermark that is designed to be as difficult to remove as possible using state-of-the-art techniques and relevant industry standards.” And it mandates that these watermarks, or other metadata attached to the AI-generated media, also contain provenance data, defined as:

Information about the history of the content, including, but not limited to, the following:

(1) The name of the generative AI provider or the camera or recording device manufacturer.

(2) The name and version number of the AI system that generated the content or the operating system, version of the operating system, or the application used to capture, create, or record the content.

(3) The time and date of the content’s creation and any additional modifications of the content.

(4) The portions of content that have been changed by a generative AI system, if applicable.

This mandate would go into effect on July 1, 2026, so there is (thankfully) time for the industry to develop better standards. But the requirement of a “maximally indelible” watermark means that complying with the bill is a constantly moving target. What if the industry converges on a standard by July 2026, and then in January 2027, some new solution is discovered? Does the entire industry have to move to the new solution, even if it has not matured into a full-blown technical standard yet? By what time do they have to transition? What if the new standard has privacy or usability implications?

AB 3211 is silent about these issues. Instead it holds the AI industry to a single, exceptionally high standard: make watermarks as difficult to modify “as possible,” regardless of other tradeoffs involved in doing so. This single-minded focus on only one objective is not generally how complex systems are engineered, and it is likely to lead to bad outcomes for AI.

Overall, the bill could make the problem of AI deepfakes and other deceptive media worse, by creating a false sense of security about what is and is not synthetically generated media. A malicious user can ultimately remove any watermark applied to AI-generated outputs if they choose to (at least as of today). They can then upload the content to social media and claim that it is authentic media. The deceptive media would then be labeled as authentic, presumably with a pleasant green badge. In that sense, AB 3211 could actually help bad actors while burdening good actors.

The details

The bill applies to every single generative AI system distributed in California, regardless of size, purpose, or who created it. It does not matter if you are a grad student making a small model to predict DNA sequences or a trillion-dollar company making a generalist multimodal model. Every piece of digital content generated by an AI system will need to be watermarked.

It is possible that Assemblymember Wicks does not realize that generative models can be trained to produce DNA sequences and did not intend for her bill to apply to those modalities. But apply they do. Of course, DNA is “just text,” yet the standard way of watermarking text described above would not apply well to DNA, lest we arbitrarily modify the genome of a novel organism being created in a lab in the interest of watermarking it.

As currently written, the bill also forbids platforms from hosting any “system” that does not have watermarking standards built into it. Here’s the relevant text:

Generative AI hosting platforms shall not make available a generative AI system that does not place maximally indelible watermarks communicating provenance data into content created or substantially modified by the system in a manner consistent with specifications set forth in paragraph (1) of subdivision (a).

And “generative AI hosting platform” is defined as “an online repository or other internet website that makes generative AI systems available for download.”

A big question here is whether an AI model counts as an “AI system” under the law. The bill doesn’t define “AI system” or mention models, but provisions like this certainly make it sound like AI models are systems, since that’s what’s typically available for download from AI hosting platforms. And that could mean that sites like HuggingFace would have to pull down generative models that don’t include AI watermarking features—which is most of them.

AB 3211 also requires generative AI system makers (again, everyone from an individual to a company—there are no thresholds here) to have their systems proactively identify themselves as AI. I support laws of this kind generally, but once again AB 3211 takes this to an extreme. In addition to requiring proactive disclosure that a given system is AI-powered, it also requires affirmative consent from the user—every single time it is used. From the bill:

In all conversational interfaces of a conversational AI system, the conversational AI system shall, at the beginning of a user’s interaction with the system, obtain a user’s affirmative consent acknowledging that the user has been informed that they are interacting with a conversational AI system. A conversational AI system shall obtain a user’s affirmative consent before beginning the conversation.

This means that every time you start a new chat with ChatGPT, or ping Siri on your phone, you will have to acknowledge that you are aware that you are interacting with an AI system. I do not see how anybody benefits from this.

Finally, at least in terms of the requirements on generative AI system makers, the bill specifies that developers must keep a public record of all system outputs that could be “deceptive.” To quote again from the bill:

A generative AI system capable of producing potentially deceptive content shall generate and store, in a searchable online database in a manner that can be retrieved by a viewer of the content, a digital fingerprint of and provenance data for any piece of potentially deceptive content that they produce. This provenance shall not include personally identifiable information.

A “digital fingerprint” means a cryptographic representation of the output rather than the output itself. Still, given that almost any text a system generates—and a great deal of other media—could be “deceptive,” (the bill defines “deceptive” content as AI-generated content that could be human-generated), this means that every AI developer will need to keep a public database of many, or perhaps all, the outputs their system has ever made.

In addition to being a substantial burden on all developers, this might be impossible for open weight models to comply with, because open weight model developers have no visibility into what users are doing with their models. The bill does nothing to grapple with this fact, at least in my reading.

In addition to all this, “large online platforms,” defined as any app or website with greater than 1 million users in California, must label all user-uploaded data (including text) as synthetic, partially synthetic, nonsynthetic, nonsynthetic with minor modifications, or as having no watermark.

Any producer of any recording equipment has to include an option for users to watermark content. This presumably includes audio, photo, or video recording equipment (your iPhone, and also the webcam on your laptop, your video doorbell, etc.) but since the bill does not define “recording equipment,” this could be many things: fMRI machines record data, too. I won’t go too much into these other requirements, because this post is already long enough.

By focusing on one goal—protecting society against synthetic media—to the exclusion of all other goals, AB 3211 creates a series of headaches for everyone while only questionably improving the problem it purports to address. Sadly, that sounds about right for AI policy these days.

The bill was unanimously passed in the State Assembly and is making its way through the Senate.