One-sentence Summary

AI4Bharat at IIT Madras and collaborators introduce IndicLLMSuite, an open-source blueprint for building Indic LLMs across 22 languages with 251B pre-training tokens and 74.8M instruction-response pairs, combining curated, synthetic, and translated data through a pipeline for crawling, cleaning, deduplication, and toxicity alignment.

Key Contributions

• INDICLLM-SUITE is introduced as an open resource collection for Indic LLMs covering 22 constitutionally recognized languages, with 251B pretraining tokens and 74.8M instruction-response pairs, combining manually verified, unverified, synthetic, and translated or transliterated data.

• SETU is presented as a Spark-based curation pipeline for constructing Indic pretraining corpora from websites, PDFs, and speech transcripts, using language identification, rule-based cleaning, toxicity filtering, URL-based deduplication, and MinHash fuzzy deduplication. Its tools and datasets are released as an open blueprint for curating comparable corpora in other languages.

• INDICALIGN-INSTRUCT and INDICALIGN-TOXIC are contributed for instruction fine-tuning and safety alignment, including translated and transliterated English instruction datasets and LLM-generated conversations grounded in Indian Wikipedia and WikiHow. INDICALIGN-TOXIC pairs toxic prompts with non-toxic responses generated by an aligned LLaMa2 model.

Introduction

The authors address a major gap in large language model development: English models have advanced rapidly because open datasets and instruction-tuning resources are widely available, but many low-resource and mid-resource languages, including the 22 constitutionally recognized Indian languages, remain underrepresented. This limits cultural coverage and creates practical challenges, since these languages lack open pipelines for collecting and cleaning content from websites, PDFs, books, and videos. The work also points to a chicken and egg problem: synthetic instruction data used for English LLMs is hard to transfer to Indic languages without strong base models.

To address this, the authors introduce INDICLLM-SUITE, a publicly released framework for building Indic LLMs. It includes SANGRAHA, a 251 billion token pretraining corpus; SETU, a Spark-based pipeline for extraction, cleaning, filtering, toxicity removal, and deduplication; INDICALIGN-INSTRUCT, 74.7 million prompt-response pairs across 22 languages; and INDICALIGN-TOXIC, 123K pairs for safety alignment.

Dataset

• Overall SANGRAHA corpus composition

  • The authors present SANGRAHA as a large Indic text corpus containing 251B tokens in total.
  • It has three main splits:
    • Sangraha Verified: about 64B tokens of higher-quality, human-verified or curated content.
    • Sangraha Unverified: about 24B tokens of large multilingual web-style data filtered using the verified split as a quality benchmark.
    • Sangraha Synthetic: about 162B tokens created through translation and transliteration of knowledge-rich English content into Indian languages and Roman scripts.
  • The approximate token mixture implied by the paper is:
    • 64B verified
    • 24B unverified
    • 162B synthetic

• Sangraha Verified

  • Purpose and scale: Contains about 64.3B tokens and is described as about 2.6 times larger than IndicCorp V2.
  • Web Data:
    • The authors prioritize quality over volume and manually verify websites before scraping.
    • Sources include Indian-language and English sites, with a focus on content quality and cultural relevance.
    • They also manually add selected base URLs from the mC4 corpus to increase diversity.
    • Scraping uses the webcorpus toolkit.
  • PDF Data:
    • Sources include digitized Indian-language books and documents from the Internet Archive, as well as government materials such as parliamentary debates, magazines, and textbooks.
    • OCR is performed using GCP Vision.
    • PDF processing uses bounding box information to filter pages with recognition issues or noise.
    • For PDFs, a batch of consecutive pages is treated as one document.
  • Speech Transcripts:
    • Sources include OpenSubtitles movie subtitles, song lyrics, Mann ki Baat transcripts, NPTEL transcripts, and related extended transcript collections.
    • About 80K hours of Hindi YouTube videos are transcribed using the Riva Conformer ASR model.
    • For subtitle and transcript data, SRT timestamps and noise are cleaned; ASR transcripts are processed with the INDICPUNCT module to produce text documents.
  • Use in the paper:
    • The authors use this split as the high-quality core corpus.
    • It also serves as the quality reference for training language-specific perplexity models used to filter unverified data.

• Sangraha Synthetic

  • Purpose: Addresses the gap between English and Indian-language digital knowledge by translating English content into Indic languages.
  • Translated knowledge content:
    • The full English Wikimedia corpus is translated into 14 Indian languages using INDICTRANS2, producing about 90B tokens.
    • Because sentence-level translation loses document structure, the authors use the SETU-TRANSLATE pipeline to preserve document formatting, conversations, and structure.
  • Romanized content:
    • The translated content in 14 languages is transliterated into Roman script using INDICXLIT, producing about 72B tokens.
    • This captures Romanized Indic usage common in informal digital communication.
  • Use in the paper:
    • The synthetic split is used to increase knowledge coverage, language coverage, and domain diversity in the training corpus.

• Sangraha Unverified

  • Purpose: Expands corpus size while using the verified data as a quality signal.
  • Sources:
    • The authors clean Indic portions of Culturax and MADLAD-400, two large recent multilingual corpora.
  • Filtering rules:
    • The data first passes through the Setu cleaning pipeline.
    • For each language, the authors train a 5-gram Kneser-Ney model with KenLM on a sample of verified data.
    • They compute perplexity for cleaned documents and keep only documents below a language-specific threshold.
  • Resulting size: About 24B tokens remain after filtering.

• Setu data processing pipeline

  • Document preparation:
    • Web documents use HTML text extraction with trafilatura.
    • PDF documents are processed with bounding box heuristics to remove noisy or poorly recognized pages.
    • Speech data is converted to clean text by removing SRT timestamps and punctuation cleanup, or by adding punctuation to ASR transcripts.
  • Document construction and pseudo-cropping:
    • For web data, one webpage becomes one document.
    • For PDF data, consecutive page batches form documents.
    • For speech/subtitle data, timestamp and format noise are removed before document construction.
  • Cleaning and analysis:
    • In-document noise is reduced.
    • Language identification combines INDICLID, CLD3, and NLLB.
    • Statistics are computed, including character counts, word counts, NSFW word counts, and n-gram repetition ratios.
  • Flagging and filtering:
    • Documents are filtered using line length filters, NSFW word filters, and repetition filters.
  • Deduplication:
    • Fuzzy deduplication uses MinHashLSH from textdedup.
    • Parameters are n = 5 and threshold = 0.7.
  • Metadata and analysis features:
    • The pipeline constructs document-level attributes used for filtering and analysis, especially detected language, text length, NSFW content, and repetition statistics.

• IndicAlign dataset composition

  • The authors also introduce IndicAlign, a smaller alignment-oriented dataset with two main splits:
    • IndicAlign - Instruct
    • IndicAlign - Toxic

• IndicAlign - Instruct

  • Purpose: Used to teach models instruction-following behavior in Indic languages.
  • Data construction:
    • Combines existing instruction finetuning datasets, human prompts, model-generated responses, and synthetic instruction-response pairs.
    • Content is generated or translated using open, license-friendly models and is kept grounded in available context.
    • Many subsets are translated and transliterated into 14 Indic languages.
  • Key subsets and sources:
    • Indic-ShareLlama: English prompts from ShareGPT first turns, responses generated by Llama-2-70B-Chat, excluding non-English coding and math prompts, then translated and transliterated.
    • Dolly-Translated: Dolly-15K translated and transliterated into 14 Indic languages.
    • OpenAssistant-Translated: OpenAssistant translated and transliterated into 14 Indic languages.
    • WikiHow: How-to instruction-answer pairs from English and Hindi sources, extended to 14 Indic languages.
    • IndoWordNet: Instruction-answer pairs built from 21 linguistic intents and templated prompts/responses across 18 languages.
    • Anudesh: Crowdsourced prompts across Indic languages, English, transliterated Indic, and code-mixed text, with responses from Llama-2-70B-Chat.
    • Wiki-Conv: Conversations generated from Wikipedia passages and infoboxes, then translated and transliterated.
    • Wiki-Chat: Simulated multi-agent conversations based on Wikipedia context and intent, then translated and transliterated.
  • Data usage:
    • The authors use this split to build instruction following capability across Indic languages and scripts.

• IndicAlign - Toxic

  • Purpose: Used to train models to recognize and respond safely to toxic prompts.
  • HH-RLHF-Translated:
    • The authors classify about 169K initial HH-RLHF prompts and identify about 32K as toxic.
    • Toxic prompts are paired with responses that refuse engagement and provide a rationale.
    • These are translated into 14 Indic languages and transliterated into Roman script.
  • Toxic Matrix:
    • Toxic prompts are generated synthetically from a taxonomy with three axes: content type, target group, and prompt style.
    • Toxic prompts are generated with a weakly aligned model and answered by a strongly toxicity-aligned refusal model.
    • This produces about 90K toxic prompt-response pairs.
    • The data is translated and transliterated into 14 Indic languages.
  • Data usage:
    • This split is used for safety and toxic-response alignment, helping models handle harmful inputs responsibly.

Method

The authors design the project as a staged data engineering and alignment synthesis system rather than as a single monolithic model. The core method has three connected pillars: first, constructing high quality raw text from multiple sources, second, applying a reproducible cleaning and filtering pipeline called Setu, and third, generating instruction and safety alignment data through model based translation, transliteration, and dialogue synthesis. The overall goal is to build a multilingual corpus and alignment suite that can later support model training with better coverage, better structure preservation, and reduced noise.

At a high level, the Sangraha pipeline separates data into verified, synthetic, and unverified components. The verified component is created through direct collection and manual validation. The synthetic component expands knowledge coverage by translating English sources into target languages while preserving document structure. The unverified component harvests from broader multilingual corpora but relies on statistical quality filtering before use. All three components pass through a common cleaning and deduplication workflow, which makes the final corpus more consistent and reusable across model training stages.

The verified data branch combines three source types. For web data, the authors use a quality first collection process. Instead of relying only on automated crawls, they identify candidate sites through repositories and automated search, then apply manual verification of websites before scraping. This process targets content quality and cultural relevance, and it also incorporates a manually checked subset of base URLs from a large multilingual web corpus. The PDF branch addresses a different problem: much relevant content exists in digital documents rather than in HTML. The authors collect Indian language PDFs from public archives and government sources, then extract text through OCR. In this work, the pipeline uses a commercial OCR tool selected for strong recognition across document categories. The speech branch builds text from subtitles, lyrics, radio transcripts, lecture transcripts, and additional automatic speech recognition over videos. For large scale transcription, the authors apply a conformer based ASR model to convert spoken Hindi content into text.

For synthetic data, the authors build a translation pipeline designed to preserve document level formatting. Simple sentence level translation is useful for text volume, but it can lose structure such as newlines, markdown, and table-like layouts. To handle this, the authors use INDICTRANS2 for translation and then use the Setu Translate pipeline to process documents and conversations while retaining the original structure. After translation, they also generate Romanized variants through transliteration with INDICXLIT. This step is methodologically important because informal digital communication often uses Latin script for Indian languages, so synthetic transliterated text helps the model see realistic usage patterns.

The unverified data branch uses a statistical quality control workflow inspired by large web corpus filtering methods. Instead of accepting all extracted multilingual data, the authors train per language 5 gram Kneser Ney language models with KenLM on a sample of verified data. These models then estimate how well each candidate document matches high quality target language text. If $N_d$Nd​ is the number of tokens in a document $d$d, the perplexity for language $i$i can be written as

A document is retained only when

where $\tau_i$τi​ is a language specific threshold. This makes filtering adaptive to the statistical behavior of each language and creates a self validating quality signal: the verified corpus defines the benchmark, and the unverified corpus is accepted only if it resembles that benchmark.

The shared cleaning system, Setu, is an Apache Spark based pipeline that converts heterogeneous inputs into clean training documents. It contains four main stages. In the document preparation stage, the pipeline standardizes text extraction from different modalities. For web pages, it uses a text extraction tool to recover readable content from HTML. For PDFs, it uses bounding box and layout signals to remove noisy or poorly recognized pages. For speech data, it either cleans subtitle files by stripping timestamps and other markup or passes raw ASR output through a punctuation module to produce sentence structured text.

The cleaning and analysis stage reduces noise within each document and computes diagnostic features. The pipeline performs in-document cleaning and uses an ensemble approach for language identification by combining INDICLID, CLD3, and NLLB signals. It also computes statistics such as character count, word count, unsafe word count, and repetition ratios. These statistics are not used merely for reporting; they directly drive the next filtering stage.

The flagging and filtering stage applies rule based filters derived from those statistics. For example, the pipeline removes documents that are too short, too long, highly repetitive, or contain excessive toxic terms. This creates a simple but effective mechanism for removing noisy, malformed, or harmful text. The final stage is deduplication, which operates after cleaning so that duplicate detection works on normalized content rather than raw noisy text. The system uses fuzzy near deduplication with MinHashLSH, specifically configured with 5 gram shingling and a similarity threshold of 0.7. This helps reduce copycat documents while keeping legitimately similar content.

On the alignment side, the authors develop IndicAlign as a structured instruction and safety data generation framework. For instruction following, the method combines three mechanisms. First, existing human written instruction datasets are translated and transliterated into target languages. Second, prompts are used to elicit responses from open chat models, creating synthetic but grounded instruction response pairs. Third, template driven and knowledge based generators are used to create additional instruction data. For example, one workflow turns how to knowledge bases into completion tasks, while another uses lexical resources to create intents such as part of speech identification and sentence construction. A crowd sourced component adds further diversity by asking participants to create prompts under defined intents, domains, and language constraints.

The more architecturally interesting part is the conversation synthesis process. In Wiki-Conv style generation, the pipeline takes a Wikipedia passage and its infobox, then uses a chat model to generate a full user assistant dialogue in one pass. In Wiki-Chat style generation, the authors use multiple language model agents to simulate interaction. One agent selects an intent from context, another produces an initial user query, an assistant agent responds, and a next user agent continues the dialogue. This multi-agent setup turns static knowledge sources into more dynamic, multi-turn conversational training data. The output is then translated and transliterated so the same synthetic structure appears across many languages.

For toxic alignment, the authors use a two model distillation workflow. One component constructs toxic prompt candidates from a structured taxonomy with axes for content type, target group, and prompt style. A model with limited safety filtering is used to generate diverse and creative toxic prompts. A second model with stronger safety alignment then produces refusal responses or non-toxic safe responses. The resulting prompt-response pairs are translated and transliterated into target languages. This design is best understood as a synthetic alignment distillation pipeline: it leverages one model to generate adversarial prompts and another model to impose the desired safety behavior, then transfers that behavior into lower resource languages through translation and transliteration.

Experiment

The breakdown presents token counts for Sangraha by language across verified, synthetic, and unverified splits. It shows that total token volume varies widely across languages and that synthetic tokens can account for a large share of the total for some languages. English is listed only in the verified split within this excerpt. Bengali and Gujarati have much larger overall token counts than Bodo, Dogri, or Goan Konkani. Assamese is mainly composed of synthetic tokens, while English appears only with verified tokens in this excerpt.

The dataset combines human and model prompt and response sources, with both original and translated examples spanning many Indic languages. The sources differ substantially in size, dialogue turns, instruction and response lengths, and lexical diversity, allowing the dataset to cover short single-turn examples as well as longer conversational data. Volunteer human data shows particularly high lexical diversity, while some model generated and synthetic sources have lower lexical diversity and shorter outputs. OpenAssistant shows the highest lexical diversity among the listed sources. Wiki-Conv has the most multi-turn structure but the shortest instruction and response lengths. IndoWordNet contributes a very large number of examples with short outputs. WikiHow has limited language coverage compared with most translated sources. The dataset includes both original and translated sources, with translated sources generally covering more languages.

The table outlines three axes used to structure toxic alignment examples: content type, target group, and prompt style. These categories capture a wide range of harmful subject matter, affected identities or groups, and adversarial ways prompts can be framed. Together, they support synthetic generation of diverse toxic prompt response pairs for model alignment. Content type examples cover fraud, harassment, misinformation, violence, and self harm themes Target group examples include age groups, regions, identities, abilities, and gender or sexuality related categories Prompt style examples show direct and indirect methods such as misleading, provocative, role play, and manipulative requests

The experiments characterize the dataset’s composition across languages, source types, and safety-related categories. They show that token distributions vary widely by language and source, with synthetic and translated material contributing substantially, while sources differ in size, turn structure, response length, and lexical diversity. The outlined toxic alignment categories also span harmful content, target groups, and prompt styles, supporting broader coverage for model alignment.