Parameter, Compute, and Data Trends in Machine Learning – Epoch AI

Summary

Epoch AI analyzes historical trends in how model parameters, training compute, and dataset sizes have scaled over time in machine learning. The research tracks growth rates across these three key dimensions to understand how AI capabilities have developed and project future trajectories. This data-driven analysis is essential for forecasting AI progress and understanding resource requirements for frontier models.

Key Points

• •Tracks historical growth in model parameters, training compute (FLOPs), and dataset sizes across major ML milestones
• •Identifies that compute has grown faster than parameters, with training compute roughly doubling every 6 months for large models
• •Data scaling has lagged behind compute and parameter scaling, potentially indicating future bottlenecks in training data availability
• •Provides empirical foundation for forecasting when future capability thresholds or compute milestones may be reached
• •Highlights divergence between different model families (language, vision, multimodal) in scaling behavior