a16z Podcast - California's Senate Bill 1047: What You Need to Know

Episode Date: June 6, 2024

On May 21, the California Senate passed bill 1047.This bill – which sets out to regulate AI at the model level – wasn’t garnering much attention, until it slid through an overwhelming bipartisan vote of 32 to 1 and is now queued for an assembly vote in August that would cement it into law. In this episode, a16z General Partner Anjney Midha and Venture Editor Derrick Harris breakdown everything the tech community needs to know about SB-1047.This bill really is the tip of the iceberg, with over 600 new pieces of AI legislation swirling in the United States. So if you care about one of the most important technologies of our generation and America’s ability to continue leading the charge here, we encourage you to read the bill and spread the word.Read the bill: https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202320240SB1047

The cost to reach any given benchmark of reasoning of capability is dropping by about 50 times every five years. The definitions for what was dangerous in the Cold War became obsolete so fast that a couple of decades later when the Macintosh launched, it was technically a munition. Great technologies always find their way into downstream uses that the original developers would have had no way of knowing about prior to launch. No rational startup founder or academic researcher is going to risk jail time or financial ruin just to advance the state of the art in AI. There's no chance we'd be here without open source. The state of California ranks as the fifth largest economy in the world. And on a per capita basis, the Golden State jumps all the way up to number two.

Now, one of the drivers of those impressive numbers is, of course, technology, with California being the home of all both. but one fan companies and a long, long tale of startups. But something happened recently that has the potential to dislocate the state's technical dominance and set a much more critical precedent for the nation. On May 21st, the California Senate passed Bill 1047. This bill, which sets out to regulate AI at the model level, wasn't garnering much attention until it slid through an overwhelming bipartisan vote of 32 to 1, and is now is now queued for an assembly vote in August, which, if passed, would cement it into law.

So here is what you need to know about this bill. Senate Bill 1047 is designed to apply to models trained above certain compute and cost thresholds. The bill also puts developers both civilly and even criminally liable for the downstream use or modification of their models, by requiring them to certify that their models won't enable, quote, hazardous capability. The bill even expands the definition of perjury and could result in jail time. Third, the bill would result in a new frontier model division, a new regulatory agency funded by the fees and fines on AI developers. And this very agency would set safety standards and advise on AI laws. Now, if all of this sounds new to you, you're not alone. But today you

have the opportunity to hear from A16Z general partner, Osheng Mita, and venture editor Derek Harris. Together, they break down everything the tech community needs to know right now, including the compute threshold of 10 to the power of 26 flops being targeted by this bill, whether a static threshold can realistically even hold up to exponential trends in algorithmic efficiency in compute cost, historical precedents that we can look to for comparison, the implications of this bill on open source, and the startup ecosystem at large, and most importantly, what you can do about it. Now, this bill really is the tip of the iceberg, with over 600 new pieces of AI legislation swirling in the United States today.

So if you care about one of the most important technologies of our generation and America's ability to continue leading the charge here, we encourage you to read the bill and spread the word. As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. Please note that A16D and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16C.com slash Disclosures. Before we dive into the substance of the California Senate Bill 1047, can you start with giving

your high-level reaction to the bill and maybe give listeners a sense of why it's such a big deal right now? Shock, disbelief. It's hard to understand just how. how blindsided startups, founders, the investor community that have been heads down building models, building useful AI products for customers, paying attention to what the state of the technology is, and ultimately just innovating at the frontier. These folks, the community broadly, feels completely blindsided by this bill. When it comes to policymaking, especially in technology at the frontier, the spirit of policymaking should be to sit down with your constituents, startups, founders, at the frontier builders, and then go solicit their opinion.

And what is so concerning about it right now is that this bill, SB 1047, was passed in the California Senate with a 32 to 1 overwhelming vote, bipartisan support. And now it's headed to an assembly vote in August, less than 90 days away, which would turn it into law. And so if it passes in California, it will set the precedent in other states. it will set a nationwide precedent, and ultimately that will have rippling consequences outside of the U.S. to other allies and other countries that look to America for guidance and for thought leadership. And so what is happening here is this butterfly effect with

huge consequences on the state of innovation. There's a lot to get into with the proposed law and some of its shortcomings or oversights. But the place I want to start is both SB 1047 and President Biden's executive order from last year, established mandatory reporting requirements for models that are trained, and this is a little difficult to speak, bear with me, listeners, that are trained on 10 to the 26 integer floating point operations per second or flops as the acronym of compute power. So can you explain to listeners what flops are and why they're significant in this context?

Right. So flops in this context refers to the number of floating point operations used to train an AI model. And floating point operations are just a type of mathematical operation that computers perform on real numbers as opposed to just integers. And the amount of flops used is a rough measure of the computing resources and complexity that went into training a model. And so if models are like cars, flops might be the amount of steel used to make a car, to borrow an analogy. It doesn't really tell you much about what the car can and cannot do directly, but it's just one way to kind of measure the difference between the steel required to. to make a sedan versus a truck. And this 10 to the 26 flop threshold is significant because that's how the bill is trying

to define what a covered model is. It's an attempt to define the scale at which AI models become potentially dangerous or in need of additional oversight. And this all starts from the premise that foundation models trained with this immense amount of computation are extremely large and capable to the point where they could pose social risks or harm inherently, if not developed carefully. But tying regulations to some fixed flop count or equivalent today is completely flawed because algorithmic efficiency improves, computing cost decline, and so models that take

far fewer resources than 10 to the 26 flops will match the capabilities of a 10 to the 26 flop model of today within a fairly short time frame. So this threshold would quickly expand to cover many more models than just the largest most cutting-edge ones being developed by tech giants. It will basically cover most startups and open source too within a really short amount of time. And so while today in 2024, realistically, only a handful of the very largest language models like GPT4 or Gemini and other top models from the big tech companies are likely to sit above that 10 to the 26 flop threshold, in reality, most open source and academic models will soon be covered by that definition

as well. This would really hurt startups. It would burden small developers, and ironically, it's going to reduce the transparency and collaboration around AI safety by discouraging open source development. What we see frequently is people in labs going out there and saying, we're going to build big state-of-the-art models that cost less to train, that use fewer resources, that use more data, or different types of data. There are all these different knobs to pull to get performance out of these models. It seems like you could have this sort of performance for a fraction of the cost and a small number of years. Right.

So that all comes down to two key trends. One, the falling cost of compute and two, the rapid progress in algorithmic efficiency. Empirically, the cost per flop for GPUs is having roughly every two to two and a half years. And so this means that a model

that costs about $100 million to train today would only cost about $25 million in about five years. And less than $6 million in a decade, just based on hardware trends alone, just Moore's Law. But that's not even the whole story, right? Algorithmic progress is also making it dramatically easier

to achieve the same benchmark performance with way less compute rapidly. And so when you look at those trends, we observe that the compute required to reach a given benchmark of reasoning or capability is decreasing by half about every 14 months or less. So if it takes 100 million worth of flops to reach some given benchmark today,

in five years, it would only take around $6 million worth the flops to achieve that same result, just considering the algorithmic progress alone. Now, when you put these two trends together, it paints a pretty stunning picture because the cost to reach any given benchmark of reasoning of capability is dropping by about 50 times every five years. And so that means that if a model costs $100 million to train to some benchmark in 2024, by 2029, it will probably cost less than $2 million. That's well within a startup budget. and by 2034, a decade, that cost will drop to somewhere between $40,000, putting it within

the reach of literally millions of people. And despite these clear trends, the advocates for the bill seem to be overlooking or underestimating this rapid progress. Some folks are suggesting that, oh, these smaller companies might take 30 years or more to reach this 10 to the 26 flop threshold. But as we've just discussed, that's a pretty serious overestimation. So even assuming a model costs a billion dollars to train to that level today, it's going to cost as little as 400,000 in just a decade. This is easily within the range for most small businesses who are going to then have to grapple with compliance and regulation and so on.

And so, look, the bottom line is that given the breakneck base of progress and compute costs and efficiency, we can expect smaller companies and academic institutions to start hitting these benchmarks in the very near future. Yeah, I think it's a relevant touchpoint to remind people that a smartphone today, like an iPhone 15 has more flops, more performance than a supercomputer did about 20 years ago. Like the world's fastest supercomputers, your iPhone can do more flops than that. The Apple Macintosh G4, I think back in 1999, had enough computing power that it would have been regulated as a national security threat. So these numbers are very much sliding scales to your point. That's right. That's right. That's a great historical example. I think there was this 1979 export administration act, right, that the U.S. had written in the Cold War era in the 70s, and the definitions for what was dangerous in the Cold War became obsolete so fast that a couple of decades later when the Macintosh launched, it was technically a munition. And so we've been here before, and we know that when policymakers and regulators try to capture the state of a current technology that's dramatically improving really fast, they become obsolete, incredibly fast.

and that's exactly what's happening here. The other thing is, at the time we're recording this, there are some proposed amendments floating around to SB 1047, one of which would limit the scope of the bill to applying, again, only to models trained at that compute capacity. And additionally, that also cost more than $100 million to train. So what's your thought on that? And again, if we attach a dollar amount to this,

doesn't it make the compute threshold kind of obsolete? Yeah, so this $100 million amendment to train might seem like a reasonable compromise at first, but when you really look at it, it has the same fundamental flaws as the original flop threshold. The core issue is that both approaches are trying to regulate the model layer itself, rather than focusing on the malicious applications or misuse of the models. Generative AI is still super early, and we don't even have clear definitions for what should be included when calculating these training costs. Do you include the data set acquisition, the researcher salaries?

Should we include the cost of previous training runs or just the final ones? Should human feedback for model alignment expenses count? If you fine-tune someone else's model, should the costs of the base model be included? These are all open questions without clear answers. And forcing startups, founders, academics to provide legislative definitions for these various cost components at this stage would place a massive burden on these smaller teams. many of whom just don't have the resources to navigate these super complex regulatory requirements. Plus, when you just look at the rapid pace of model engineering,

these definitions would need to be updated constantly, which would be a major drain on innovation. So when you combine that ambiguity with the criminal and monetary liabilities proposed in the bill, as well as the broad authority they're trying to give to the new frontier model division, which is sort of like a DMV for AI models that they're proposing, which can arbitrarily decide these matters, the outcome is clear, right? Most startups will simply have to relocate to more AI-friendly states or countries, while open-source AI research in the U.S. will be completely crushed due to the

legal risks involved. So in essence, the bill is creating this disastrous, regressive tax on AI innovation. Large tech companies that have armies of lawyers and lobbyists will be able to shape the definitions to their advantage, while smaller companies, open-source researchers and academics will be completely left out in the cold. It's almost like saying we've just invented the printing press, and now we're only going to let those folks who can afford $100 million budgets to make these printing presses decide what can and cannot be printed.

It's just blatant regulatory capture, and it's one of the most anti-competitive proposals I've seen in a long time. And what we should be focusing on instead is regulating specific, high-risk applications and malicious end users. That's the key to ensuring that it, AI benefits everyone, not just a few. Now, you've mentioned that the purported goal of some of these bills, 1047 in particular,

is to prevent against what you might call catastrophic harms or existential risks from artificial intelligence. But I'm curious, do you think, I mean, are the biggest threats from LLMs really weapons of mass destruction or bio-weapons or autonomously carrying out criminal behavior? I mean, if we're going to regulate these models, I mean, should we not regulate use cases that are like proven in the wild and can actually do real damage today versus hypothetically at some point these things could happen? Absolutely. I mean, basically what we have

is a complete over-rotation of the legislative community around entirely non-existent concerns of what is being labeled as AI safety when what we should be focusing on is AI security. These models are no different than databases or tools in the past that have given humans more efficiency, better ways to express themselves. They're really just

neutral pieces of technology. Now, sure, they may be increasing or allowing bad actors to increase the speed and scale of the attacks, but the fundamental attack vectors remain the same. It's spearfishing, deep fakes, it's misinformation. And these attack vectors are known to us. And we should focus on how to strengthen enforcement and give our country better tools. to enforce those laws in the wake of increasing speed and scale of these attacks. But the attacks themselves, the attack vectors haven't changed. It's not like AI suddenly has exposed us to tons of new ways to be attacked. And that's just so far off.

And frankly, unclear, and today largely in the realm of science fiction. And so the safety debate often centers around what is called existential risk or these models autonomously going rogue to produce weapons of mass destruction, or the Terminator SkyNet situation where they're hiding their true intentions from us and sure, maybe there's some theoretically tiny likelihood that happens many, many, many, many years from now, but exceptional claims require exceptional evidence.

And so the real threat here is from us not focusing on the misuses and malicious users of these models and putting the burden of actually doing that on startups, on founders, and engineers. Right. And to your point, even if a model made it marginally easy, easier to learn, let's say, how to build a bioweapon. Like, one, people know how to do that today.

We have all of these things. We have labs dedicated to all of these things. You still need to get materials to carry out these attacks. And there are regulations around acquiring those materials and databases around who's buying one. Yes, it does seem like the existing legal framework for some of these major threats is very robust. Exactly. I think what we really need is more investment in defensive artificial intelligence solutions. What we need is to arm our country, our defense departments, our enforcement agencies with the tools they need to keep up with the speed and scale at which these attacks are being perpetuated, not slowing down the fundamental innovation that can actually unlock those defensive applications.

And look, the reality is America and her allies are up against a pretty stiff battle from adversarial countries around the world who aren't stopping their speed of innovation. And so it's almost an asymmetric warfare against ourselves that's being proposed by SB 1047. Yeah, I'm certain there are governments that would, in fact, fund those $100 million models. It's well north of that, right? Yeah. And we have increasing evidence that this is happening

and that our national security actually depends on improving and accelerating open source collaboration. So just two months ago, the Department of Justice revealed and published a public investigation, the conclusion of which was that a Google engineer was boarding a plane to China with a thumb drive, with Frontier AI Hardware Schematics from Google. This was a nation-state-sponsored attack on our AI ecosystem.

And the only defense we have against that is actually making sure that innovation continues at breakneck speed in the country, not adding more burden to model innovation. The other thing that SB 1047 would do, which we haven't really touched on, is imposed liability, civil and, in some cases, criminal liability,

on model developers for the civil liability part. If they build a model that's covered by this bill, they need to be able to prove with beyond reasonable assurance or whatever the language is that this could not possibly be used for any of these types of attacks. And also they have to be able to prove that no one else could come along

and say fine-tune their model and use it for some sort of attack, right? So that's a whole new level to be on the hook for money as an individual or jail time as an individual for building this model and not making it, quote-unquote, safe enough. Oh, no, you're absolutely right. The idea of imposing civil and criminal liability on model developers,

when downstream users do something bad, is so misguided and such a dangerous precedent. First off, the bill requires developers to prove that their models can't possibly be used for any of the defined hazardous capabilities. But as we just discussed, these definitions are way too vague, ambiguous, and subject to interpretation. How can a developer prove a negative, especially when the goalposts keep moving?

It's an impossible standard to meet. Second, the bill holds developers responsible for any misuse of their models, even if that misuse comes from someone else who's fine-tuned or modified the model. It's ridiculous. It's like holding car manufacturers liable for every accident caused by a driver who's modified their car. So it's an absurd standard that no other industry has held to. The practical effect of these liability provisions will be to drive AI development underground or offshore. No rational startup founder or academic researcher is going to risk jail time or financial ruin

just to advance the state of the art in AI. They'll simply move their operations to a jurisdiction with a more sensible regulatory environment and the U.S. will lose out, period. The worst part, these liability provisions actually make us less safe, not more. By driving AI development into the shadows, you lose the transparency and open collab that's essential. for identifying and battle-hardening vulnerabilities in AI models. What we need is more open-source development, not less. So while the build sponsors may have good intentions,

imposing blanket liability on model developers for hypothetical future misuse is the exact opposite of what we need. Right. Supporters might argue, well, we put some behind bars for lying to the government for lying about the capabilities of their models. But again, like, you might not know the capabilities of your models, right, or what a downstream user could do with that. And I wanted to ask you, too, because you've built startups.

You invest in startups. I mean, can you walk through, like, the kind of wrench this type of compliance would throw into whether it's the finances or the operation or just the general way that startups and innovative companies work? Oh, yeah. Look, I love California. And that's why I'm fighting so hard for this. I did my undergraduate and graduate work here in the Bay.

I founded my first company here. I sold that to another California company. And over the last decade plus that I've been here, it's only become more and more clear to me that a huge part of what makes the entire startup AI ecosystem even work is the ability for founders to take bold technology risks without having to worry about the kinds of ambiguity and liability risks that this bill is proposing.

When we first started Ubiquity 6, my last company, the goal was to empower developers to use our computer vision pipeline for all kinds of new use cases that we hadn't even imagined. We had some idea of what people would do, originally augmented reality applications, but after we launched it, we found millions of users who used our 3D mapping technology for entirely new kinds of users from architecture and robotics to VFX and entertainment that we hadn't even considered. And so the whole engine and the beauty of platform businesses is that developers can focus on

developing general and highly flexible technology and then just let the market figure out entirely new niche use cases at scale. And this is true of almost every great AI business I've either worked with directly or invested in, right, whether it was mid-journey and image generation and anthropic and language models or 11 labs and audio models. Great technologies always find their way into downstream users that the original developers would have had no way of knowing about prior to launch. And to burden that process with the liability of this bill of saying that developers have to somehow, prior to launch, demonstrate beyond any shred of reasonable doubt, which is, again, a completely ambiguous definition in the bill that these users were

known about, their risks were understood, that exhaustive safety testing had been done to make sure none of these things would be possible, just completely kill that engine. If we went back in time, and this bill passed as currently envisioned, as much as I hate to say it, there's no chance I would have founded my company in California. Speaking of startups, that's to say nothing about open source projects and open source development, which have been like a huge driver of innovation over the past couple of decades. We're talking about very, very bootstrapped skeletal budgets on some of these things, but hugely, hugely important.

Oh, fundamentally, I don't think the current wave of modern. generative scaling laws-based AI would even exist without open source, right? If you just go back and look at how we got here, its formers, kind of the atomic unit of how these models learn, was an open source widely collaborated on development, right? In fact, it was produced at one lab, Google, and allowed another lab after open publishing and collaboration, which was open AI, to actually continue that work. And there's no chance we'd be here without open source. The downstream contributions of open source continue to be massive today. When when a company like Mistral or Facebook open source models and release the weights,

that allows other startups to then pick up on their investments and build on top of them. It's like having the Linux to the closed source Windows operating systems. It's like having the Android to the closed source iOS. And without those, there's no chance that the speed at which the AI revolution is moving will continue. Certainly not in California and probably not in the United States. Open source is kind of the heart of software innovation. and this bill slows it down, has a chilling effect on open source

by putting liability on the researchers and the builders pushing open source forward. Yes, and the other thing about open source is, I guess this is true of any model theoretically, but the idea someone takes it and builds on it, right? In generative AI or foundation models, you would call that fine-tuning, right? Where you kind of retrain a model

to your own purposes, using your own data. And again, this bill would, as written, imposed liabilities, again, on the original developers if someone is able to fine-tune their model to perform theoretically some sort of bad act, right? I mean, how realistic is it for someone to even build a model that would be resistant or resilient against these types of fine-tuning attacks or optimizations, for lack of a better term?

Yes, so this is another can of worms as well. Again, a symptom of the root cause of this bill's flawed premise of regulating models instead of misuses. So in the current bill draft, the language says that these restrictions and regulations will extend to a concept of a derivative model, which is a model that is a modified version of another model, such as a fine-tuned model.

So if someone makes a derivative model of my base model that's harmful, I am now liable for it. It's akin to saying that if I'm a car manufacturer and someone turns a car I made into a tank by putting guns on it and shoots people with it, I should get thrown in jail. The definition of what a derivative model is also super vague. And so now the bill sponsors are considering an amendment that says, oh, let's add a compute cap to this definition.

And they've decided to pick 25%, which is quite arbitrary, and to say if somebody uses more than 25% of the compute that the base model developer used to fine tune a model, then it's no longer a derivative model. And you're off the hook for it as the base model developer. Well, that's absolutely nonsensical as well. As some great researchers like Eon Stoica at Berkeley have shown, it takes an extremely small amount of compute to fine-tune a model like Vicuna, where with just 70,000 shared GPT conversations, they fine-tuned Lama to become one of the best open-source models at the time, showing it really doesn't take much computer data to turn a car into a tank to borrow an analogy. And so like with the 10 to the 26 compute threshold issue we discussed earlier, this is just another arbitrary. magic number, the bill authors are pulling out of thin air, to try and define model layer computing dynamics that are so early and changing that it's absolute over-regulation and will kill the speed of innovation here.

All right. So you've alluded to this, but I wanted to ask directly, if we say not all regulation is bad, if you were in charge of regulating AI, how would you approach it or how would you advise lawmakers who feel compelled to address what seemed like concerns over AI? What would be your approach? non-negotiable really here should be zero liability at the model layer. What you want to do is target misuses and malicious users of AI models, not the underlying models and not the infrastructure. And that's the core battle here. I think that's the fundamental flaw of this bill

is it's trying to regulate the model and infrastructure and not instead focus on the misuses and malicious users of these models. And so over time, I think it would prove out that the right way to keep the U.S. at the frontier of responsible, of secure AI innovation, is to actually focus on the malicious users and misuses of models,

not slow down the model and infrastructure layer. We should focus on concrete AI security and strengthening our enforcement and our defenses against AI security attacks that are increasing at speed and scale, but fundamentally, these safety concerns that are largely science fiction and theoretical are a complete distraction at the moment. And lastly, we have no choice, but to absolutely accelerate open source innovation, we should be investing in open source collaboration between

America and our allies to keep our national competitiveness from falling behind our adversarial countries. And so the three big policy principles I would look for from regulators would be to regulate and focus and target misuses, not models, to prioritize AI security over safety and to accelerate open source. But the current legislation is absolutely prioritizing the wrong things and is rooted in a bunch of arbitrary technical definitions that will be outmoded, obsolete, and overreaching fairly soon.

One might say we should regulate it the same way we regulate the Internet, just to say, let it thrive. It really is tantamount to saying, we've barely just invented the printing press, We've barely just invented the Model T Ford car. And now what we should immediately do is try to rush and prevent future improvements to cars or to the printing press

by largely putting the responsibility of any accidents that happened from people irresponsibly driving the car out on the streets on Henry Ford or of the inventors of the printing press. So then the final question here, taking everything into account, what can everyday listeners do about this? I mean, if I'm a founder, if I'm an engineer,

or if I'm just concerned, what can I do to voice my opinion about SB 1047, about frankly, any regulation coming down the line? How should people think about making their voice heard? Yeah, so I think three steps here. The first would be to just read the bill. It's not very long, which is good. But most people just haven't had a chance to actually read it. Stem two, especially for people in California,

the most effective way to have this bill be opposed is for each listener to call their assembly rep and tell them why they should vote no on this bill in August, right? This is less than 90 days away. So we really don't have much time for all of the assembly members to hear just how little support this bill has from the startup community, tech founders, academics. And step three is to go online, you know, make your voice heard on places like Twitter where it turns out, you know, a lot of both state level and national level legislators

do listen to people's opinions. And so, look, I think if this bill passes, in California, it sure as hell is going to create a ripple effect throughout other states, and then this will be a national battle. If you liked this episode, if you made it this far, help us grow the show. Share with a friend, or if you're feeling really ambitious, you can leave us a review at rate thispodcast.com slash A16c. You know, candidly, producing a podcast can sometimes feel like you're just talking into a void

And so if you did like this episode, if you liked any of our episodes, please let us know. I'll see you next time.