Dolma 3 (Ai2 Open Corpus)
Technical audit of Ai2's 5.2T+ token Dolma 3 pre-training mix covering ODC-By provenance, MinHash decontamination, PDF parsing, and commercial limits.
Open Dolma 3 (Ai2 Open Corpus) DossierDATASET & HUB AUDIT INDEX
AI Training Dataset, Open Corpus & Hub Reviews (2026 Edition)
Independent 2026 technical reviews of 14 foundation AI datasets, hubs, and licensed exchanges (Dolma 3, FineWeb, Nemotron-CC, DROID, Hugging Face, Protege).
Answer in brief:This index audits 14 foundational AI training corpora, open distribution hubs, and licensed data exchanges across token scale, deduplication methodology, SPDX copyright lineage, and benchmark contamination risk. While permissive open releases like Dolma 3, FineWeb, and DROID anchor base model pre-training, commercial enterprises hit hard license and reasoning ceilings on scraped data. Every dossier below maps exact technical limits and bridges into verified, indemnity-cleared human post-training labs.
Rank: 14 Audited Corpora & HubsBest for: Vetting pre-training corpora, cloud exchanges & multimodal licensingLicense / Neutrality: SPDX, ODC-By, CC0, CC-BY-NC & Commercial DSA AuditsPricing: Free open Parquet shards to $50K–$500K+ licensed archivesScale / Turnaround: Up to 30T+ web tokens & 1M+ embodied trajectories
Individual Dataset, Hub & Exchange Technical Audits
FineWeb, FineWeb-Edu & FineWeb2
Engineering review of Hugging Face FineWeb (15T), FineWeb-Edu, and FineWeb2 (1,000+ languages): Datatrove dedup rules, Parquet layout, and limits.
Open FineWeb, FineWeb-Edu & FineWeb2 DossierNemotron-CC (v2.1)
Technical review of NVIDIA Nemotron-CC (6.3T tokens): ensemble classifier scoring, 1.9T synthetic rephrasing tokens, license terms, and pre-training gains.
Open Nemotron-CC (v2.1) DossierCommon Crawl (WARC / WAT / WET)
Technical evaluation of raw Common Crawl archives (250B+ pages): why WET text hurts LLM quality, WARC extraction pipelines, PII exposure, and copyright risks.
Open Common Crawl (WARC / WAT / WET) DossierRedPajama-Data-v2
Technical review of Together AI's RedPajama-Data-v2 (30T+ raw tokens, 5 languages): pre-computed quality annotations, MinHash signatures, and filtering trade-offs.
Open RedPajama-Data-v2 DossierLAION-BVD (Big Video Dataset Ecosystem)
Audit of LAION-BVD multimodal video-text indices: URL-only WebDataset hydration, >25% link rot, copyright barriers, and licensed commercial video alternatives.
Open LAION-BVD (Big Video Dataset Ecosystem) DossierNVIDIA Granary (Multilingual Speech Corpus)
Technical audit of NVIDIA Granary (1M+ hours across 25 languages): NeMo Speech Data Processor pseudo-labeling, ASR/AST accuracy, and commercial voice limits.
Open NVIDIA Granary (Multilingual Speech Corpus) DossierAgiBot World (Embodied Robotics Corpus)
Technical audit of AgiBot World (1M+ bimanual manipulation trajectories, 100+ robots): sensor synchronization, RLDS/LeRobot schemas, and CC-BY-NC-SA limits.
Open AgiBot World (Embodied Robotics Corpus) DossierDROID (Distributed Robot Interaction Dataset)
Technical review of DROID (76K trajectories, 564 real-world scenes, CC-BY-4.0): stereo camera calibration, Franka kinematics, and commercial VLA limits.
Open DROID (Distributed Robot Interaction Dataset) DossierHugging Face Datasets & Hub
Technical audit of Hugging Face Datasets Hub (250K+ repos): Arrow/Parquet streaming, Xet storage, Croissant metadata, license traps, and contamination risks.
Open Hugging Face Datasets & Hub DossierKaggle Datasets & Competition Corpora
Technical review of Kaggle Datasets (350K+ uploads): high-signal competition benchmarks (ARC-Prize, AI Math Olympiad), CSV pitfalls, and commercial IP risks.
Open Kaggle Datasets & Competition Corpora DossierAWS Data Exchange (ADX)
Technical review of AWS Data Exchange for AI training corpora: direct S3 access, IAM governance, licensed financial/clinical feeds, and AI training addenda.
Open AWS Data Exchange (ADX) DossierSnowflake Marketplace
Technical audit of Snowflake Marketplace for enterprise AI: zero-copy Secure Data Sharing, Cortex Knowledge Extensions, clean rooms, and GPU export limits.
Open Snowflake Marketplace DossierProtege (Licensed Multimodal & Clinical Data Exchange)
Independent audit of Protege (withprotege.ai): rights-cleared de-identified EHR/DICOM medical datasets, licensed audiovisual libraries, and AI training rights.
Open Protege (Licensed Multimodal & Clinical Data Exchange) DossierOpen & Gated Foundation Dataset Benchmark Table (2026 Edition)
Audit 36 canonical pre-training, SFT, code, speech, vision, and embodied robotics corpora across SPDX license lineage, commercial production clearance, and enterprise saturation boundaries. Export full structured records via /data.json or /data.csv.
Showing 36 of 36 audited foundation corpora
| Dataset & Modality | Scale (Tokens / Hours / Trajectories) | Top-Level License | Commercial Clearance | Primary Enterprise Limitation |
|---|---|---|---|---|
| Dolma 3 Mix Allen Institute for AI (Ai2) · Pre-Training TextRead Full Audit → | 5.2T+ tokens (web, academic PDFs, code, math) | ODC-By 1.0 (upstream source terms apply) | Mixed | ODC-By wrapper covers curation, not underlying publisher copyright or robots.txt retro-revocations. |
| FineWeb (v1.3) Hugging Face · Pre-Training TextRead Full Audit → | 15T+ tokens (96+ Common Crawl snapshots) | ODC-By 1.0 | Mixed | Web-scraped HTML retains third-party copyright claims in EU DSM Art. 4 opt-out audits. |
| FineWeb-Edu Hugging Face · Pre-Training TextRead Full Audit → | 1.3T & 5.4T educational-filtered tokens | ODC-By 1.0 (Llama-3 classifier annotations) | Mixed | Classifier bias over-weights textbook prose while starving conversational, tabular, and edge-case reasoning. |
| FineWeb2 (Multilingual) Hugging Face · Pre-Training TextRead Full Audit → | 8T+ tokens across 1,000+ languages | ODC-By 1.0 | Mixed | Low-resource language shards exhibit severe OCR/machine-translation boilerplate contamination. |
| FinePDFs ↗ Hugging Face · Pre-Training Text | 1.8T+ tokens parsed from scientific/technical PDFs | ODC-By 1.0 (document-level rights vary) | Mixed | Multi-column table linearization errors and embedded publisher All-Rights-Reserved PDFs. |
| Nemotron-CC (v2.1) NVIDIA · Pre-Training TextRead Full Audit → | 6.3T tokens (4.4T real + 1.9T synthetic rephrased) | NVIDIA Open Model / Common Crawl Terms | Mixed | Synthetic rephrased splits carry upstream generator license notices and web provenance obligations. |
| Common Corpus ↗ PleIAs / Cultural Heritage Consortium · Pre-Training Text | 2T+ tokens (public domain & permissive open texts) | Public Domain / CC-BY / CC0 | Yes | Skews heavily historical/archival; lacks modern post-2024 API frameworks and contemporary syntax. |
| Common Crawl Raw WARC/WET Common Crawl Foundation · Pre-Training TextRead Full Audit → | 250B+ web pages (~400+ TiB per crawl) | CC Terms of Use (raw crawl access only) | Mixed | Unfiltered PII, SEO spam, adult content, and zero copyright indemnity without 6-stage deduplication. |
| RedPajama-Data-v2 Together AI · Pre-Training TextRead Full Audit → | 30T+ raw tokens w/ 40+ quality signals across 5 languages | Apache 2.0 scripts / Common Crawl upstream | Mixed | Delivers uncleaned indices + quality metadata; teams must run their own MinHash and PII scrubbing. |
| TxT360 ↗ LLM360 · Pre-Training Text | 15T+ tokens (globally deduplicated across 99 CC dumps + ArXiv/FreeLaw) | ODC-By 1.0 | Mixed | Cross-corpus global deduplication still leaves benchmark paraphrase overlap against MMLU/GPQA. |
| DCLM-Baseline (DataComp-LM) ↗ DataComp-LM Consortium · Pre-Training Text | 4T filtered tokens from 240T pool | CC-BY-4.0 metadata / Common Crawl upstream | Mixed | fastText OH-2.5 + ELI5 classifier biases output toward Q&A forum cadence. |
| CulturaX ↗ uonlp · Pre-Training Text | 6.3T tokens across 167 languages (mC4 + OSCAR) | mC4 / OSCAR upstream terms | Mixed | OSCAR registration gates and heterogeneous national copyright regimes across 167 locales. |
| Llama-Nemotron Post-Training ↗ NVIDIA · Post-Training SFT / Pref | 30M+ SFT & reasoning traces (math, code, tool-use) | CC-BY-4.0 + Llama / DeepSeek / Qwen upstream clauses | Training-Only | Contains distilled synthetic outputs subject to upstream model-family naming and acceptable-use clauses. |
| Dolci-Instruct-SFT (Olmo 3) ↗ Allen Institute for AI (Ai2) · Post-Training SFT / Pref | 1.8M+ curated multi-turn SFT & reasoning dialogues | ODC-By 1.0 (sub-mixture licenses vary) | Mixed | Certain synthetic sub-splits inherit non-competing or attribution clauses from teacher models. |
| Tulu 3 SFT Mixture ↗ Allen Institute for AI (Ai2) · Post-Training SFT / Pref | 939K SFT prompts & completions across 18 sources | ODC-By 1.0 (composite upstream licenses) | Mixed | Includes WildChat/ShareGPT-style & GPT-distilled subsets blocked by strict enterprise legal teams. |
| HelpSteer2 & HelpSteer3 ↗ NVIDIA · Post-Training SFT / Pref | 21K–40K+ human multi-attribute preference & edit pairs | CC-BY-4.0 | Yes | Small total row volume; saturates quickly on repository-scale engineering and domain verticals. |
| UltraFeedback (Binarized) ↗ OpenBMB / Hugging Face H4 · Post-Training SFT / Pref | 64K prompts / 256K LLM-judged completions | MIT | Training-Only | GPT-4 judge annotations trigger OpenAI Terms-of-Service restrictions for competing commercial models. |
| WildChat-1M ↗ Allen Institute for AI / Academic · Post-Training SFT / Pref | 1M real user-chatbot multi-turn conversations | ODC-By 1.0 (OpenAI API upstream outputs) | Non-Commercial | Contains unredacted residual PII risk and OpenAI-generated assistant turns. |
| LMSYS-Chat-1M ↗ LMSYS Org · Post-Training SFT / Pref | 1M arena conversations across 25 LLMs | LMSYS Non-Commercial Research License | Non-Commercial | Explicitly restricted to non-commercial academic research; prohibited in commercial model weights. |
| OpenCodeReasoning ↗ NVIDIA · Code & SWE | 736K+ competitive programming chain-of-thought solutions | CC-BY-4.0 | Training-Only | Single-file algorithmic contests only; zero multi-file git repo architecture or CI integration. |
| The Stack v2 ↗ BigCode / Software Heritage · Code & SWE | 67.5B files (~900B+ tokens across 619 languages) | Per-file SPDX (permissive + opt-out governance) | Mixed | Requires downloading raw blobs from Software Heritage S3 and honoring ongoing author opt-out removals. |
| StarCoder2Data / Stack-Edu ↗ BigCode / Hugging Face · Code & SWE | 125B–900B educational & permissive code tokens | Gated BigCode OpenRAIL-M / Per-File SPDX | Mixed | Copyleft GPL/AGPL file leakage risk if strict SPDX license allow-lists are not enforced pre-training. |
| CommitPackFT ↗ BigCode · Code & SWE | 702K filtered git commit-message-to-diff pairs (277 langs) | MIT / Apache-2.0 / BSD permissive subset | Yes | Short single-commit diffs lacking multi-commit architectural refactors or executable test harnesses. |
| SWE-bench Verified (Eval Split) ↗ OpenAI / Princeton NLP · Code & SWE | 500 human-validated GitHub issue resolution tasks | MIT | Yes | Evaluation holdout ONLY (500 rows); training on it invalidates public coding benchmark integrity. |
| LAION-BVD (Big Video Dataset) LAION Ecosystem · Vision & MultimodalRead Full Audit → | 100M+ video-text clip URLs & spatio-temporal captions | CC-BY-4.0 metadata index (videos unhosted) | Non-Commercial | URL-only index pointing to copyrighted web video streams; link rot (>25%) and zero commercial video rights. |
| PixMo ↗ Allen Institute for AI (Ai2) · Vision & Multimodal | 2.3M+ dense voice-transcribed image captions, points & QA | ODC-By 1.0 | Mixed | Annotations are open, but underlying web image URLs still require commercial image-rights clearance. |
| SA-1B (Segment Anything) ↗ Meta AI · Vision & Multimodal | 11M licensed images / 1.1B segmentation masks | SA-1B Research License | Non-Commercial | Research-only license strictly forbids training commercial production foundation models. |
| Objaverse-XL ↗ Allen Institute for AI (Ai2) · Vision & Multimodal | 10.2M 3D objects & meshes (GitHub, Thingiverse, Sketchfab) | ODC-By 1.0 index (per-asset CC-BY / CC-BY-NC mix) | Mixed | Over 40% of crawled 3D meshes carry Non-Commercial or No-Derivatives creator licenses. |
| NVIDIA Granary (v2) NVIDIA NeMo · Speech & AudioRead Full Audit → | 1M+ hours multilingual ASR & AST speech (25 European langs) | CC-BY-4.0 pipeline / YouTube & VoxPopuli upstream | Mixed | Pseudo-labeled speech transcripts lack studio-grade speaker consent and paralinguistic emotion labels. |
| People's Speech ↗ MLCommons · Speech & Audio | 30,000+ hours English speech w/ CC-BY / CC-BY-SA provenance | CC-BY-4.0 / CC-BY-SA | Mixed | CC-BY-SA sub-shards trigger ShareAlike legal review; predominantly monologue civic/lecture audio. |
| Mozilla Common Voice 27.0 ↗ Mozilla Foundation · Speech & Audio | 33,000+ validated hours across 130+ languages | CC0 1.0 Public Domain Dedication | Yes | Short isolated sentence read-prompts; lacks overlapping conversational turn-taking and noisy telephony. |
| Emilia-Dataset ↗ Amphion / Open Audio · Speech & Audio | 101,000+ hours in-the-wild speech (6 languages) | CC-BY-NC-4.0 | Non-Commercial | Strict CC-BY-NC-4.0 license bars commercial TTS/voice-LLM deployment without separate rights acquisition. |
| AgiBot World (Beta / Colosseo) AgiBot Research · Robotics & EmbodiedRead Full Audit → | 1M+ bimanual manipulation trajectories (217 tasks, 100+ robots) | CC-BY-NC-SA 4.0 (Open Research Release) | Non-Commercial | CC-BY-NC-SA blocks commercial humanoid/industrial deployment without bilateral commercial licensing. |
| DROID (Distributed Robot Interaction) TRI / Stanford / Berkeley / Consortium · Robotics & EmbodiedRead Full Audit → | 76,000 teleoperated trajectories (350 hours, 564 scenes) | CC-BY-4.0 | Yes | Locked to single-arm Franka Panda + Robotiq gripper kinematics; fails to transfer directly to dexterous hands. |
| Open X-Embodiment (OXE) ↗ Google DeepMind & 34 Academic Labs · Robotics & Embodied | 1M+ episodes across 22 robot embodiments (RLDS format) | Composite Apache 2.0 / CC-BY / Custom Lab Licenses | Mixed | Heterogeneous camera intrinsics, control frequencies (3Hz–50Hz), and mixed academic sub-dataset licenses. |
| ToyBlocks (Embodied Spatial & Assembly) ↗ Embodied Vision Benchmark Consortium · Robotics & Embodied | 120K+ fine-grained 6-DoF assembly & contact-rich episodes | CC-BY-NC-4.0 | Non-Commercial | Tabletop rigid-block domain under-represents deformable objects, kitchen fluids, and industrial tolerances. |
Head-to-Head Corpus, Hub & Modality Comparisons
FineWeb vs. Dolma 3
Side-by-side technical comparison of Hugging Face FineWeb (15T) vs Ai2 Dolma 3 (5.2T): deduplication ablations, scientific PDFs, code mix, and ODC-By lineage.
Inspect Comparison MatrixCommon Crawl vs. FineWeb
Engineering comparison of raw Common Crawl WARC/WET archives vs Hugging Face FineWeb: compute cost, DOM extraction fidelity, deduplication, and when to go raw.
Inspect Comparison MatrixHugging Face Datasets vs. Kaggle Datasets
Technical comparison of Hugging Face Datasets Hub vs Kaggle Datasets: Arrow/Parquet distributed streaming, Xet deduplication, competition quality, and licensing.
Inspect Comparison MatrixAWS Data Exchange vs. Snowflake Marketplace
Enterprise comparison of AWS Data Exchange vs Snowflake Marketplace for licensed AI training & RAG datasets: S3 GPU streaming, Clean Rooms, EDP spend, and IP.
Inspect Comparison MatrixCITE THIS REFERENCE ENTRYBest AI Data Reference Registry. “AI Training Dataset, Open Corpus & Hub Reviews (2026 Edition).” Last reviewed November 2026. https://bestaidata.org/reviews/