原始内容
Thomas Wolf (@Thom_Wolf) 转发了 elie (@eliebakouch) 的帖子：
more than ~5T deduped code training tokens with a permissive license, very important release
previous versions of the stack were used in almost every model disclosing the datasets they use, this is a free upgrade for every lab https://t.co/h9tA1pwQb3
> **引用原帖 Anton Lozhkov (@anton_lozhkov):**
> For over two years, the largest open code dataset was The Stack v2… until today.
> 🥞 The Stack v3 is out: the largest open code dataset ever released: 114 TB, 770 languages, 224M repositories, ~5T tokens of deduplicated and filtered source code. Fully open, no restrictively licensed code included.
> The upgrade:
> - v2 (2024): 68 TB raw -> 2 TB / ~550B tokens, 618 languages filtered
> - v3 (2026): 114 TB raw -> 15.9 TB / ~5T tokens, 713 languages filtered
> C++ x15, TypeScript x7.5, Rust x7, Python x4.8. Even the COBOL corner of GitHub got a bigger slice.
> Part of that is two fresh years of open source. Part of it is a bug we found in v2's deduplication - story below. 👀
> Things that make v3 different:
> 1. Contents inline. The #1 complaint about v2 was "cool dataset, where's the actual code?" v2 shipped file IDs from the Software Heritage graph, and fetching contents was a DIY treasure hunt. v3 is self-contained: sources embedded directly, one row = one repository. Download finishes -> you start training.
> 2. Fresh crawl. v2 was a 2023 snapshot of even older crawls. v3 is a direct re-crawl of GitHub at the latest commit, completed by August 2025: 224M repositories, 44B files. Forks were only included if they had 5+ stars.
> Two ways in:
> 🥞 stack-v3-train - near-deduplicated, quality-filtered, PII-redacted, contents inline. Point load_dataset at it and go.
> https://t.co/px2LW3v6VB
> 🏗 stack-v3-full - the entire 114 TB corpus as an HF Storage Bucket: every duplicate kept with cluster IDs, stubs for excluded files. Roll your own dedup, filters, and mixes.
> https://t.co/SQzP5gOmzl
> The record isn't really ours to claim - it's millions of everyone's repositories, neatly stacked. Thanks to every developer who keeps their code public, the BigCode community, and friends who built the infrastructure that made this possible!
> https://x.com/anton_lozhkov/status/2080254608639701222