Large, diverse code corpora are a practical prerequisite for training and evaluating code-focused language models; this dataset supplies a very large, language-tagged dump of public GitHub files optimized for model training workflows.
What Sets It Apart
- Scale and breadth — 115 million files across 30 programming languages and 60+ extensions, so models see long-tail, multilingual code patterns rather than a handful of dominant languages.
- File-level metadata and licenses — each example includes repo name, path, inferred language, declared license and byte size, enabling targeted filters (e.g., Dockerfiles, MIT-licensed Python files) and dataset slicing for experiments.
- Streaming-first access — recommended streaming API reduces RAM/disk needs for most workflows; full download is available but requires ~300GB compressed (~1TB uncompressed).
- Simple preprocessing provenance — created from a BigQuery GitHub export with long-line and duplicate removal, making the collection reproducible and easier to reason about for large-scale training.
Who it's for and trade-offs
Great fit if you need large, diverse code corpora for pretraining, fine-tuning, or evaluating code generation/LMs and want simple language/license filters without building a scraper. Look elsewhere if you require legally curated, fully provenance-verified or non-public code (e.g., proprietary codebases), or if you need line-level context cleaning beyond the provided preprocessing. Also note the dataset can contain harmful, insecure, or sensitive snippets (credentials, vulnerabilities); downstream users should apply additional scrubbing and license checks before production use.
