NVIDIA Releases 2.8 Trillion Indic Pretraining Tokens

NVIDIA and IIT Madras lab AI4Bharat have introduced Nemotron-pretraining-Indic-v1, releasing approximately 2.8 trillion tokens across 22 Indian languages spoken by 97 percent of Indians (nearly 1.2 billion citizens in the 2011 Census). AI4Bharat head Mitesh Khapra, named to the 2025 TIME100 AI list, previously released Sangraha to address language data deficits. Measured with the Gemma-4 tokenizer, the new corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens across 34 language-script configurations. This covers native and romanized text for 12 languages and native-script text for 10 additional languages.

Built using 15 Nemotron-CC-v2 subsets, the open GPU-accelerated pipeline runs on NeMo Data Designer and Gemma-4-31B-it to translate English documents, evaluate quality against an MQM-inspired rubric, and repair low-scoring records. Keeping outputs scoring at least 75 out of 100 recovered 438.4 billion tokens. For 10 languages facing document stability challenges, the workflow incorporates Bodhan AI's Indic Translate model, using strategies like overlapping translation windows for Santali or converting through Bengali script for Manipuri.

During continued pretraining for Nemotron 3.5 Super VL, 10 billion Indic tokens were added from a 100 billion token budget (with 10 percent allocated to Indic text and 90 percent maintaining prior proportions). The stage improved performance by 4.98 to 9.82 percentage points across five benchmarks: MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K (excluding Odia). Compared against Nemotron 3 Super across overall training, Nemotron 3.5 Super VL improved on WMT24++ (measured in chrF points), five task benchmarks, and four IndiVibe preference dimensions, despite a 2.04 point dip on IFEval 1.5.

For AI practitioners, these resources simplify multilingual model development. Developers can inspect configurations like hi-Deva-IN or hi-Latn-IN using Hugging Face Datasets, generate custom synthetic data using the Nemotron Bharat translation plugin with Nemotron-Personas-India, or evaluate models using the new Indic-LLM-Eval-Suite.