Summary: We introduce a command-line tool for hardening datasets against less sophisticated scrapers.

Author: Alex Turner. Contributors: Dipika Khullar, Ed Turner, and Roy Rinberg.

Dataset contamination is bad for several reasons. Most obviously, when benchmarks are included in AI training data, those benchmarks no longer measure generalization -- the AI may have been directly taught the answers. Even more concerningly, if your data promote negative "stereotypes" about AIs, they might become self-fulfilling prophecies, training future models to exhibit those very behaviors.

In the Claude 4 system card, Anthropic revealed that approximately 250,000 transcripts from their alignment faking paper had been scraped from the public web and included in their pretraining data. This caused an early model to hallucinate details from the paper's fictional scenarios, forcing Anthropic to implement unique mitigations. Speculatively, this kind of misalignment data could degrade the alignment of any models trained thereafter.[1]

However, this result wouldn't [...]

---

Outline:

(02:11) A download portal in minutes

(02:53) What we provide

(02:56) A web portal

(03:06) A CLI tool

(03:54) Possible improvements

(05:01) Please protect datasets

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
July 25th, 2025

Source:
https://www.lesswrong.com/posts/DA3vbSEfABLdoCt59/we-built-a-tool-to-protect-your-dataset-from-simple-scrapers

---

Narrated by TYPE III AUDIO.

---

Images from the article:

At most, users click a box. They don't have to complete any annoying tasks.

Generally, you'd just download a single .zip – possibly created by our easy-dataset-share command-line tool.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“We Built a Tool to Protect Your Dataset From Simple Scrapers” by TurnTrout, Edward Turner, Dipika Khullar

LessWrong (30+ Karma)

“We Built a Tool to Protect Your Dataset From Simple Scrapers” by TurnTrout, Edward Turner, Dipika Khullar