The Stack v3 发布：最大开放代码数据集

Thomas Wolf (@Thom_Wolf) 转发了 Anton Lozhkov (@anton_lozhkov) 的帖子： For over two years, the largest open code dataset was The Stack v2… until today. 🥞 The Stack v3 is out: the largest open code dataset ever released: 114 TB, 770 languages, 224M repositories, ~5T tokens of deduplicated and filtered source code. Fully open, no restrictively licensed code included. The upgrade: - v2 (2024): 68 TB raw -> 2 TB / ~550B tokens, 618 languages filtered - v3 (2026): 114 TB raw -> 15.9 TB / ~5T tokens, 713 languages filtered C++ x15, TypeScript x7.5, Rust x7, Python x4.8. Even the COBOL corner of GitHub got a bigger slice. Part of that is two fresh years of open source. Part of it is a bug we found in v2's deduplication - story below. 👀 Things that make v3 different: 1. Contents inline. The #1 complaint about v2 was "cool dataset, where's the actual code?" v2 shipped file IDs from the Software Heritage graph, and fetching contents was a DIY treasure hunt. v3 is self-contained: sources embedded directly, one row = one repository. Download finishes -> you start training. 2. Fresh crawl. v2 was a 2023 snapshot of even older crawls. v3 is a direct re-crawl of GitHub at the latest commit, completed by August 2025: 224M repositories, 44B files. Forks were only included if they had 5+ stars. Two ways in: 🥞 stack-v3-train - near-deduplicated, quality-filtered, PII-redacted, contents inline. Point load_dataset at it and go. https://t.co/px2LW3v6VB 🏗 stack-v3-full - the entire 114 TB corpus as an HF Storage Bucket: every duplicate kept with cluster IDs, stubs for excluded files. Roll your own dedup, filters, and mixes. https://t.co/SQzP5gOmzl The record isn't really ours to claim - it's millions of everyone's repositories, neatly stacked. Thanks to every developer who keeps their code public, the BigCode community, and friends who built the infrastructure that made this possible!