One-sentence Summary

Researchers from FAIR at Meta, the University of Washington, and the University of Chicago introduce the Byte Latent Transformer (BLT), a byte-level LLM that encodes bytes into dynamically sized patches segmented by next-byte entropy, and they demonstrate that it matches tokenization-based LLM performance at scale up to $8\text{B}$8B parameters and $4\text{T}$4T training bytes while delivering significant improvements in inference efficiency and robustness.

Key Contributions

• The paper introduces the Byte Latent Transformer (BLT), a byte-level LLM architecture that replaces fixed-vocabulary tokenization with dynamically sized patches segmented by next-byte entropy, allocating compute and model capacity according to data complexity.
• It presents the first FLOP-controlled scaling study of byte-level models, scaling up to 8B parameters and 4T training bytes, and shows that BLT matches tokenization-based LLM performance at this scale.
• Results demonstrate that BLT reduces inference FLOPs by up to 50% with minor evaluation trade-offs, improves robustness to noisy input and sub-word handling, and enables a joint scaling of model and patch size within a fixed inference budget.

Introduction

Existing large language models rely on tokenization, a heuristic pre-processing step that groups bytes into fixed tokens. This approach introduces biases that cause sensitivity to input noise, poor orthographic knowledge, and uneven performance across languages. Directly training on raw bytes has been impractical because of long sequence lengths, and prior efforts using efficient attention or attention-free architectures failed to address the dominant computational cost of feed-forward layers, which run on every byte.

The authors introduce the Byte Latent Transformer (BLT), a tokenizer-free architecture that learns directly from raw bytes. BLT dynamically groups bytes into patches using a learnable method, with lightweight encoder and decoder modules mapping arbitrary byte groups into latent representations. In a FLOP-controlled scaling study up to 8B parameters and 4T training bytes, BLT matches the performance of tokenization-based models like Llama 3 while using up to 50% fewer FLOPs at inference. It also shows improved robustness to noisy inputs and better character-level understanding, and unlocks a new scaling dimension where model and patch size can both increase within a fixed inference budget.

Dataset

The authors use a controlled experimental setup to compare BLT against tokenization-based models, ensuring no advantages are given to BLT, particularly regarding sequence context length. The dataset construction and usage are described below.

Pre-training Datasets

• BLT-Exp: A dataset containing 2 trillion tokens sourced from a variety of cleaned and filtered public sources. It is used for ablation studies and scaling trend experiments to determine the optimal architectural choices for BLT.
• BLT-1T: A higher quality 1 trillion token dataset gathered from public sources, which also includes a subset of pre-training data released by Datacomp-LM. This dataset is used for the full pre-training run to compare BLT against the Llama 3 architecture on downstream tasks.

Entropy Model and Threshold

• For models using entropy patching, the authors estimate a patching threshold to achieve a desired average patch size on the pre-training data mix. They default to a global entropy threshold when not specified.
• The entropy model is a transformer with 100M parameters, 14 layers, and a hidden dimensionality of 512. It uses sliding window attention of 512 bytes and is trained on the same data distribution as BLT.
• If the receptive field of the entropy model is small enough, it can be encoded as an efficient lookup table.

Efficient Training on Patches

• Dynamic patching methods result in different byte-to-patch ratios when loading batches. The implementation packs batches of patches instead of bytes to maintain a constant number of patches per batch and avoid padding steps in the latent transformer.
• Byte sequences are padded or truncated to 12k for BLT-Exp and 24k for BLT-1T to prevent memory spikes caused by unusually large patches.

Equalizing Context Length

• To ensure no model has an advantage from a larger context, the number of bytes in each batch remains constant on average. The sequence length of the Latent Transformer is adjusted to achieve 8k and 16k byte contexts for BLT-Exp and BLT-1T, respectively.

Method

The authors introduce a method that segments byte sequences into variable-sized patches to dynamically allocate computational resources based on context. This approach contrasts with fixed-size tokenization by grouping bytes according to information density. The segmentation is governed by a patching function that identifies boundaries where the model's uncertainty is highest. Specifically, the method employs an entropy-based strategy that computes next-byte entropies to determine where to split the sequence. The entropy is calculated as:

Boundaries are identified either by a global entropy threshold or by detecting points that break approximate monotonicity in entropy. As shown in the figure below, various segmentation heuristics, including strided, space-based, and entropy-driven methods, produce distinct patch structures that influence the computational cost.

The overall system architecture comprises three primary modules: a lightweight local encoder, a large latent global transformer, and a lightweight local decoder. Refer to the framework diagram for the data flow through these components. The local encoder maps a sequence of raw input bytes into expressive patch representations. These patch representations are then processed by the latent global transformer, which performs the autoregressive prediction and consumes the majority of the floating-point operations. Finally, the local decoder maps the global patch representations back into raw byte sequences. This design allows the model to vary the amount of compute expended for different portions of the input sequence.

The local encoder is designed to efficiently aggregate byte-level information into patch-level representations. Input bytes are embedded and augmented with hash n-gram embeddings to capture context from preceding bytes. The core aggregation mechanism utilizes a series of alternating transformer layers and cross-attention modules. In the encoder cross-attention, query vectors correspond to the patch representations, while keys and values are derived from the byte hidden states. This allows the model to pool byte information specifically relevant to each patch boundary. The queries are initialized by pooling the byte representations corresponding to each patch and passing them through a linear projection. The update rule for the cross-attention layer is defined as:

where $Q$Q represents the patch queries and $K, V$K,V represent the byte keys and values. A masking strategy ensures that each patch query only attends to the keys and values of the bytes belonging to that specific patch.

The local decoder mirrors the encoder's architecture but reverses the information flow to reconstruct bytes from patches. It takes the global patch representations as input and predicts the underlying byte sequence. The decoder utilizes a cross-attention mechanism where the roles of queries and key-values are interchanged. Byte representations serve as queries, and patch representations serve as keys and values. This allows the decoder to unpatch the global representations back into fine-grained byte predictions. The decoder applies alternating cross-attention and transformer layers to refine the byte representations before generating the final output, effectively mapping the latent patch space back to the byte vocabulary.

Experiment

Experiments compared BLT against BPE tokenizer based models under controlled, FLOP matched conditions with equalized context lengths to avoid giving BLT any advantage. BLT matched or outperformed BPE baselines in compute optimal scaling trends, achieved better downstream task performance beyond compute optimal and with larger patch sizes, and showed strong gains on robustness benchmarks including noisy data, character understanding, and low-resource machine translation. Ablations identified decoder cross-attention and n-gram hash embeddings as key contributors to performance, with hash gains diminishing after 300K hashes.

FLOP-matched 8B models show that BLT with monotonic entropy patching outperforms a Llama 3 tokenizer-based model on the majority of tasks, while space patching trades accuracy for significant inference FLOP savings. Global entropy patching matches the baseline overall but lags on structured tasks like MMLU. Monotonic entropy patching beats the Llama 3 baseline on 4 out of 7 tasks, including large gains on ARC-E and HumanEval. Space patching underperforms on most tasks but reduces inference FLOPs via a larger average patch size of 6 bytes versus 4.5. Global entropy patching slightly trails the baseline on MMLU while matching or exceeding it on other tasks like HellaSwag and MBPP.

BLT demonstrates strong robustness to byte-level noise, matching or exceeding a larger Llama 3.1 model on average across noisy HellaSwag variants while using fewer training bytes. It also shows clear advantages over tokenizer-based models on character-level and phonology tasks. On noisy HellaSwag, BLT ties Llama 3.1 on average and outperforms it on repeat, random case, and uppercase corruption. BLT exceeds the BPE Llama 3 baseline on grapheme-to-phoneme conversion and character understanding benchmarks.

BLT outperforms the Llama 3 BPE model on FLORES-101 translation tasks in both directions, with a larger advantage when translating into English. The gains are particularly pronounced for low-resource languages, highlighting byte-level modeling's effectiveness for long-tail language families. BLT achieves higher BLEU than Llama 3 for both language-to-English and English-to-language translation directions. The largest relative improvements over the BPE baseline occur in low-resource language settings. BLT's overall average advantage is about 2 BLEU points for translation into English and 0.5 points for translation from English.

Cross-attention in the decoder is the primary contributor to improved language modeling performance, with the best results achieved when it is applied across all layers. Encoder cross-attention only yields a slight improvement when queries are initialized via pooling of encoder hidden states. The gains are most pronounced on Common-Crawl data, suggesting cross-attention helps in diverse or noisy domains. Decoder cross-attention consistently lowers bits-per-byte, especially when applied at all layers. Encoder cross-attention improves performance only when combined with pooling initialization of queries. Cross-attention provides the largest benefit on Common-Crawl, with smaller gains on Wikipedia and GitHub.

Ablation experiments show that adding n-gram hash embeddings substantially improves bits-per-byte across all domains, with the strongest gains on Wikipedia and GitHub. Smaller n-gram sizes (3,4,5) are more effective than larger ones, and using a smaller per-ngram vocabulary with varied n-gram sizes outperforms larger vocabularies of the same total size. Hash n-gram embeddings yield large BPB improvements over no embeddings, particularly on Wikipedia and GitHub. N-gram sizes 3,4,5 outperform larger sizes 6,7,8, indicating shorter context n-grams are more impactful. For a fixed total vocabulary, using smaller per-ngram vocabularies with distinct n-gram sizes performs better than larger uniform vocabularies.

The evaluation shows that byte-level models with monotonic entropy patching outperform a Llama 3 tokenizer-based baseline on most tasks, while space patching trades accuracy for inference FLOP savings. BLT also demonstrates strong robustness to byte-level noise, matching or exceeding a larger Llama 3.1 model on noisy benchmarks, and achieves clear gains over BPE models on character-level tasks and low-resource language translation. Ablations reveal that decoder cross-attention is the main driver of improved language modeling, and adding n-gram hash embeddings with smaller n-gram sizes yields substantial bits-per-byte gains across domains, especially on Wikipedia and GitHub.