Jack Clark, Anthropic

# [Import AI](https://jack-clark.net/ "Home")

### What does 10^25 versus 10^26 mean?

#### by Jack Clark

_A brief look at what FLOPs-based regulation nets out to_

Recent AI regulations have defined the trigger points for oversight in terms of the amount of floating point operations dumped into training an AI system. If you’re in America and you’ve trained a model with 10^26 FLOPs, you’re going to spend a lot of time dealing with government agencies. If you’re in Europe and you’ve trained a model with 10^25 FLOPs, you’re going to spend a lot of time dealing with government agencies.

**More details:**

In the United States, the recent Biden Executive Order on AI says that general-purpose systems trained with 10^26 FLOPs (or ones predominantly trained on biological sequence data and using a quantity of computing power greater than 10^23) fall under a new reporting requirement that means companies will let the US government know about these systems and also show work on testing these systems.

In Europe, the recent EU AI Act says that general-purpose systems trained with 10^25 FLOPs have the potential for “systemic risk” and that people who develop these models “are therefore mandated to assess and mitigate risks, report serious incidents, conduct state-of-the-art tests and model evaluations, ensure cybersecurity and provide information on the energy consumption of their models.”

Given how difficult the task of assessing AI systems is, these thresholds matter – governments will need to staff up people who can interpret the results about models which pass these thresholds.

**What is the difference between 10^25 versus 10^26 FLOPs in terms of money?**

Let’s say you wanted to train an AI system – how much money would you spend on the compute for training the system before you hit one of these thresholds? We can work this out:

**NVIDIA H100 – NVIDIA’s latest GPU.**

_Assumptions:_

Using FP8 precision – various frontier labs (e.g, [Inflection](https://inflection.ai/inflection-2)) have trained using FP8

40% efficiency – assuming you’ve worked hard to make your training process efficient. E.g., Google claims [~46% for PALM 540B](https://arxiv.org/pdf/2204.02311.pdf)

$2 per chip hour – assuming bulk discounts from economies-of-scale.

Training a standard Transformer-based, large generative model.

**10^26**

Flops per chip second = 2000e12\* × 0.4 = 8E14

Flops per chip hour = flops per chip s × 60 (seconds per minute) × 60 (minutes per hour) = 2.88E18

chip h = 1e26 / flops per chip h = 34.722M

chip h × $2 = **$69.444M**

\*3958 TFLOPS (for fp8 with sparsity) on [H100 SXM](https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet) divided by 2 (because the 2x sparsity support generally isn’t relevant for training), so the right number is 1979e12. But the datasheet doesn’t have enough information to tell you that; you just have to know!

**10^25**

Flops per chip second = 2000e12 × 0.4 = 8E14

Flops per chip hour = flops per

... (truncated, 31 KB total)