The Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, or SB 1047, was a failed 2024 California bill intended to "mitigate the risk of catastrophic harms from AI models so advanced that they are not yet known to exist". Specifically, the bill would have applied to models which cost more than $100 million to train and were trained using a quantity of computing power greater than 1026 integer or floating-point operations. SB 1047 would have applied to all AI companies doing business in California—the location of the company would not matter. The bill would have created protections for whistleblowers and required developers to perform risk assessments of their models prior to release, with guidance from the Government Operations Agency. It would also have established CalCompute, a University of California public cloud computing cluster for startups, researchers and community groups. == Background == The rapid increase in capabilities of AI systems in the 2020s, including the release of ChatGPT in November 2022, caused some researchers and members of the public to become concerned about the existential risks associated with increasingly powerful AI systems. Hundreds of tech executives and AI researchers, including two of the so-called "Godfathers of AI", Geoffrey Hinton and Yoshua Bengio, signed a statement in May 2023 calling for the mitigation of the "risk of extinction from AI" to be a global priority alongside "pandemics and nuclear war". However, the plausibility of these risks is still widely debated. Strong regulation of AI has been criticized for purportedly causing regulatory capture by large AI companies like OpenAI, a phenomenon in which regulation advances the interest of larger companies at the expense of smaller competition and the public in general, although OpenAI ended up opposing the bill. Other advocates of AI regulation aim to prevent bias and privacy violations, rather than existential risks. For example, some experts who view existential concerns as overblown and unrealistic view them as a distraction from near-term harms of AI like discriminatory automated decision making. In the face of existential concerns, technology companies have made voluntary commitments to conduct safety testing, for example at the AI Safety Summit and AI Seoul Summit. In 2023, not long before the bill was proposed, Governor Newsom of California and President Biden issued executive orders on artificial intelligence. State Senator Wiener said SB 1047 draws heavily on the Biden executive order, and is motivated by the absence of unified federal legislation on AI safety. Historically, California has passed regulation on several tech issues itself, including consumer privacy and net neutrality, in the absence of action by Congress. == History == === Proposal and voting === The bill was authored by State Senator Scott Wiener. Wiener first proposed AI legislation for California through an intent bill called SB 294, the Safety in Artificial Intelligence Act, in September 2023. On February 7, 2024, Wiener introduced SB 1047. On May 21, SB 1047 passed the Senate 32–1. The bill was significantly amended by Wiener on August 15, 2024, in response to industry advice. Amendments included adding clarifications, and removing the creation of a "Frontier Model Division" and the penalty of perjury. On August 28, the bill passed the State Assembly 48–16. Then, due to the amendments, the bill was once again voted on by the Senate, passing 30–9. === Veto by governor === On September 29, Governor Gavin Newsom vetoed the bill. The deadline for California lawmakers to overrule Newsom's veto was November 30, 2024. Newsom cited concerns over the bill's regulatory framework targeting only large AI models based on their computational size, while not taking into account whether the models are deployed in high-risk environments. Newsom emphasized that this approach could create a false sense of security, overlooking smaller models that might present equally significant risks. He acknowledged the need for AI safety protocols but stressed the importance of adaptability in regulation as AI technology continues to evolve rapidly. Governor Newsom also committed to working with technology experts, federal partners, and research institutions, including the Carnegie Endowment for International Peace, led by former California Supreme Court Justice Mariano-Florentino Cuéllar; and Stanford University's Human-Centered AI (HAI) Institute, led by Dr. Fei-Fei Li. He announced plans to collaborate with these entities to advance responsible AI development, aiming to protect the public while fostering innovation. == Provisions == SB 1047 would have covered AI models with training compute over 1026 integer or floating-point operations and a cost of over $100 million. If a covered model is fine-tuned using more than $10 million, the resulting model would also have been covered. The bill would have defined critical harms with respect to four categories: Creation or use of a chemical, biological, radiological, or nuclear weapon Cyberattacks on critical infrastructure causing mass casualties or at least $500 million of damage Autonomous crimes causing mass casualties or at least $500 million of damage Other harms of comparable severity Developers would have needed to create a "safety and security protocol" before training covered models. Before deployment, they would have submitted a statement of compliance, confirming they took reasonable care to take measures to prevent covered models that pose an unreasonable risk of critical harms. The statement would have included risk assessments and descriptions of their compliance process. These rules would have applied to both covered models and their derivatives, including post-training modifications, with annual third-party audits required starting in 2026. Safeguards to reduce risk included the ability to shut down the model, which has been variously described as a "kill switch" and "circuit breaker". Whistleblowing provisions would have protected employees who report safety problems and incidents. Additionally, SB 1047 would have created a public cloud computing cluster called CalCompute, associated with the University of California, to support startups, researchers, and community groups that lack large-scale computing resources. === Compliance and supervision === SB 1047 would have required developers, beginning January 1, 2026, to annually retain a third-party auditor to perform an independent audit of compliance with the requirements of the bill, as provided. The Government Operations Agency would have reviewed the results of safety tests and incidents, and issue guidance, standards, and best practices. The bill would have created a Board of Frontier Models to supervise the application of the bill by the Government Operations Agency. It is would be composed of 9 members. == Reception == === Subjects of debate === Proponents of the bill described its provisions as simple and narrowly focused, with Sen. Scott Weiner describing it as a "light-touch, basic safety bill". This was disputed by critics of the bill, who described the bill's language as vague and criticized it as consolidating power in the largest AI companies at the expense of smaller ones. Proponents, in turn, argued that the bill only applies to models trained using more than 1026 FLOPS and with over $100 million, or fine-tuned with more than $10 million, and that the threshold could be increased if needed. The penalty of perjury was also a subject of debate, and was eventually removed through an amendment. The scope of the "kill switch" requirement was also reduced, following concerns from open-source developers. The use of the term "reasonable assurance" in the bill was also controversial, and it was eventually amended to "reasonable care". Critics then argued that "reasonable care" imposed an excessive burden by requiring confidence that models could not be used to cause catastrophic harm; proponents claimed that the standard did not require certainty and that it already applied to AI developers under existing law. === Support and opposition === Individual supporters of the bill included Turing Award recipients Yoshua Bengio and Geoffrey Hinton, Elon Musk, Bill de Blasio, Kevin Esvelt, Dan Hendrycks, Vitalik Buterin, OpenAI whistleblowers Daniel Kokotajlo and William Saunders, Lawrence Lessig, Sneha Revanur, Stuart Russell, Jan Leike, actors Mark Ruffalo, Sean Astin, and Rosie Perez, Scott Aaronson, and Max Tegmark. Over 120 Hollywood celebrities, including Mark Hamill, Jane Fonda, and J. J. Abrams, also signed a statement in support of the bill. Max Tegmark likened the bill's focus on holding companies responsible for the harms caused by their models to the FDA requiring clinical trials before a company can release a drug to the market. Organizations sponsoring the bill included the Center for AI Safety, Economic Security California and Encode. The la
Read more →
