Back
Stanford HAI framework
webcrfm.stanford.edu·crfm.stanford.edu/open-fms/
Relevant to debates around open-source AI safety: whether releasing model weights openly increases risk from misuse or improves safety through broader scrutiny; produced by Stanford HAI's CRFM group, a leading academic institution in foundation model research.
Metadata
Importance: 58/100homepageanalysis
Summary
Stanford's Center for Research on Foundation Models (CRFM) presents a framework for evaluating and governing open foundation models, addressing the tradeoffs between openness and safety in large AI systems. It provides structured analysis of how open release policies affect safety, accountability, and beneficial use of foundation models.
Key Points
- •Examines the tradeoffs of open vs. closed release strategies for large foundation models and their implications for safety and misuse
- •Provides a governance framework for thinking about when and how foundation models should be released openly
- •Addresses accountability mechanisms for foundation model developers and downstream users
- •Considers dual-use risks and the challenge of preventing harmful applications of openly released models
- •Connects technical deployment decisions to broader policy and societal impact considerations
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Open Source AI Safety | Approach | 62.0 |
Cached Content Preview
HTTP 200Fetched Mar 20, 202623 KB
# **On the Societal Impact of** **Open Foundation Models**
**Analyzing the benefits and risks of foundation models with widely available weights**
[Paper (ICML 2024 Oral)](https://arxiv.org/pdf/2403.07918v1.pdf) [Blog](https://www.aisnakeoil.com/p/on-the-societal-impact-of-open-foundation) [Policy brief](https://hai.stanford.edu/sites/default/files/2023-12/Governing-Open-Foundation-Models.pdf) [Authors](https://crfm.stanford.edu/open-fms/#authors)
**Context.** One of the biggest tech policy debates today is about the future of AI,
especially foundation models and generative AI. Should open AI models be restricted? This
question is central to
several policy efforts like the EU AI Act and the U.S. Executive Order on Safe, Secure, and
Trustworthy AI.
**Status quo.** Open foundation models, defined here as models with widely available weights,
enable greater customization and deeper inspection.
However, their downstream use cannot be monitored
or moderated. As a result, risks relating to biosecurity, cybersecurity, disinformation, and
non-consensual
deepfakes have prompted pushback.
**Contributions.** We analyze the benefits and risks of open foundation models. In
particular, we present a framework to assess their _marginal_ risk compared to closed
models or
existing technology. The framework helps explain why the marginal risk is low in some cases,
clarifies disagreements in past studies by revealing the different assumptions about risk, and
can help foster more
constructive debate going forward.
#### Key contributions
- **Identifying distinctive properties.** Foundation models released with widely available
weights have
distinctive properties that lead to both their benefits and risks. We outline five
properties
that inform our analysis of their societal impact: broader access, greater customizability, the
ability for local inference and adaptability, an inability to rescind model weights once
released, and an inability to monitor or moderate usage.
- **Connecting properties to benefits.** Open foundation models can distribute decision-making
power, reduce market
concentration,
increase innovation, accelerate science, and enable transparency.
We highlight considerations that may temper these benefits in practice (for example, model
weights
are
sufficient for some forms of science, but access to training data is necessary for others and is
not
guaranteed by release of weights).
- **Developing a risk assessment framework.** We present a framework for conceptualizing the
_marginal_
_risk_ of open foundation models:
the extent to which these models increase societal risk by intentional misuse beyond
closed foundation models or pre-existing technologies (such as web search on the internet).
- **Re-assessing past studies.** Surveying seven common misuse vectors described for open
foundation
models (such as disinformation,
biosecurity, cybersecurity, non-consensual intimate imagery, scams), we find that past studies
do
not
clearly assess the m
... (truncated, 23 KB total)Resource ID:
a2e936ba01459a44 | Stable ID: OWIwNzhjZG