Arb Research Work Portfolio

[![](https://arbresearch.com//img/arb-minimal.svg)](https://arbresearch.com/)[Arb news](https://arbresearch.com/)

[Semantic duplicates confound apparent AI progress](https://www.arxiv.org/abs/2602.12413)

_February 2026_

**Author**:  Gavin, Ari, Juan, Tomas, Peli, Nicky, Nandi


**Client**:  Pro bono



* * *

New paper studying AI performance. ( [The Twitter thread](https://x.com/g_leech_/status/2023384075537432662) is an accessible way in.)

In the past 3 years, LLM training corpuses expanded by a factor of 10000x. Does this increased chance of including test data confound apparent AI progress? What about including “semantic” duplicates, things which happen to be very similar to test data? And so how much of LLM performance is really down to “local” generalisation (pattern-matching to hard-to-detect semantically-equivalent training data)?

We experiment on OLMo 3, one of the only really good models with open training data. Since we have its entire training corpus, we can exhaustively check for real “natural” duplicates and finetune it to estimate their impact. We embed the entire training corpus.

We were surprised by how ineffective the standard n-gram decontamination is at catching exact duplicates - 70% of harder tasks had a match.

Every single MBPP test example and 78% of CodeForces have semantic duplicates.

So: n-gram decontamination is not enough even for the easy (exact) stuff, semantic duplicates are at least a moderately big deal, and this probably transfers to frontier models to some degree.

The first entry in a research programme with the grand aim of decomposing benchmark gains / apparent AI progress into 4 estimates:

1. benchmaxxing (memorising the test / paraphrasing / finetuning for a narrow task)
2. usemaxxing (RLing narrow capabilities in)
3. local generalisation (highly sophisticated pattern-matching but all bound by training data)
4. Deep OOD generalisation (e.g. algorithms that solve a whole problem class)

Obviously this will take a lot more work to nail down, and three times that much in closed models.

**[Arxiv](https://www.arxiv.org/abs/2602.12413), [code](https://github.com/AriSpiesberger/Soft-Contamination-Prevelance), [long explainer](https://x.com/g_leech_/status/2023384075537432662)**

[![](https://arbresearch.com/img/sdtd.jpg)](https://www.arxiv.org/abs/2602.12413)

* * *

* * *

[2025 review](https://arbresearch.com/work/)

_December 2025_

**Author**:  Gavin, Misha, Kristie, Juan, Tomáš, Peli, Phil, Simon, Sam, Rory, Rian, Stag, Ari, Nicky, Nandi, Paul, Stephen, Kristi, David, Lydia, Vidur, Jord, Ozzie, Olga, Lauren, Ulkar, Jooda


**Client**:  Misc



* * *

Most of our work this year was again private; we completed 37 projects with 4.8 FTE and spent 3 months colocated in Stockholm and London.

### Public highlights

- Dwarkesh and Gavin’s book finally [came out](https://arbresearch.com/work/#Stripe%20Press%20book%20out) in Stripe Press.

- We collected [200](https://frontier2025.netlify.app/) of the bigge

... (truncated, 50 KB total)