How do you know when an artificial intelligence system is so powerful that it poses a security risk and should not be deployed without careful oversight?
For regulators trying to constrain AI, it’s mostly about the math. Specifically, an AI model trained on 10 to the 26th of floating-point operations per second must now be reported to the U.S. government, and could soon trigger even stricter requirements in California.
Say what? Well, if you count the zeros, that’s 100,000,000,000,000,000,000,000,000,000, or 100 septillion, calculations per second, using a measure known as flops.
To some lawmakers and AI safety advocates, this means a level of computing power that could enable rapidly developing AI technology to create or proliferate weapons of mass destruction or launch catastrophic cyberattacks.
Those who have crafted such regulations acknowledge that they are an imperfect starting point for distinguishing today’s best-performing generative AI systems from the next generation, which could be even more powerful. These systems are largely made by California-based companies like Anthropic, Google, Meta Platforms and ChatGPT maker OpenAI.
Critics have criticized the thresholds as arbitrary: an attempt by governments to regulate mathematics.
“Ten to the 26th are flops,” venture capitalist Ben Horowitz said in a podcast this summer. “Well, what if that’s the size of the model you need to, say, cure cancer?”
An executive order signed by President Joe Biden last year relies on that threshold. So does California’s recently passed AI safety law, which Gov. Gavin Newsom has until Sept. 30 to sign or veto. California adds a second metric to the equation: Regulated AI models must also cost at least $100 million to build.
Following Biden’s lead, the European Union’s sweeping AI Act also measures floating-point operations per second, or flops, but sets the bar 10 times lower, to 10 to the 25th power. That applies to some AI systems already in use. The Chinese government has also looked at measuring computing power to determine which AI systems need security.
There are no publicly available models that meet California’s higher threshold, though it’s likely that some companies have already started building them. If so, they would have to share certain details and safeguards with the U.S. government. Biden used a Korean War-era law to force tech companies to alert the U.S. Commerce Department if they build such AI models.
AI researchers are still debating how best to evaluate the capabilities of the latest generative AI technology and how it compares to human intelligence. There are tests that assess AI on its ability to solve puzzles, reason logically, or how quickly and accurately it predicts what text will answer a human chatbot question. Those metrics help judge an AI tool’s usefulness for a given task, but there’s no easy way to know which ones will be so widely applicable that they pose a threat to humanity.
“This calculation, this flop number, is by general consensus the best we have in this area,” said physicist Anthony Aguirre, executive director of the Future of Life Institute, which has worked to pass California’s Senate Bill 1047 and other AI safety regulations around the world.
Floating point arithmetic may sound complicated, but it’s essentially just adding or multiplying numbers, Aguirre says, making it one of the simplest ways to assess the capacity and risk of an AI model.
“Most of what these things do is just multiply big tables of numbers together,” he said. “You can just think of it as punching a bunch of numbers into your calculator and adding them up or multiplying them. And that’s what it does — ten trillion times or a hundred trillion times.”
For some tech leaders, however, it’s too simplistic and hard-coded a metric. There’s “no clear scientific support” for using such metrics as a proxy for risk, computer scientist Sara Hooker, who leads the nonprofit research arm of AI company Cohere, argued in a July paper.
“The current calculation thresholds are short-sighted and unlikely to mitigate the risk,” she wrote.
Venture capitalist Horowitz and his business partner Marc Andreessen, founders of the influential Silicon Valley investment firm Andreessen Horowitz, have attacked the Biden administration and California lawmakers over AI regulations, saying they could stifle the nascent AI startup industry.
To Horowitz, setting limits on “how much math you can do” reflects a misconception that there are only a handful of large companies making the most capable models and that you can “put flaming hoops in front of them and they’ll jump through it and that’s fine.”
In response to the criticism, the sponsor of the California legislation sent a letter to Andreessen Horowitz this summer defending the bill, including its statutory thresholds.
Regulation of more than 10 to the 26th flops is “a clear way to exempt from safety testing requirements many models that we know, based on current evidence, are not capable of causing critical harm,” wrote Sen. Scott Wiener of San Francisco. Existing publicly available models “have been tested to very dangerous capabilities and are not subject to the law,” Wiener said.
Both Wiener and Biden’s executive order consider the metric a temporary measurement that can be adjusted later.
Yacine Jernite, who leads policy research at the AI firm Hugging Face, said the flops metric was created in good faith prior to last year’s Biden order but is already starting to show its age. AI developers are doing more with smaller models that require less processing power, while the potential harms of more widely deployed AI products won’t trigger the scrutiny California is proposing.
“Some models have a much greater impact on society and should be subject to higher standards. Other models are more exploratory in nature and it may not be useful to use the same kind of process to certify them,” Jernite said.
Aguirre says it makes sense for regulators to be flexible, but he sees some of the resistance to the flop threshold as an attempt to circumvent regulation of AI systems as they become more powerful.
“This is all happening very quickly,” Aguirre said. “I think there’s a legitimate criticism that these thresholds don’t capture exactly what we want them to capture. But I think it’s a weak argument to go from that to, ‘Well, we should just do nothing and just cross our fingers and hope for the best.'”