LLM Tokenizer Comparison: BPE vs SentencePiece (Unigram and SP‑BPE)

Overview

Large language models live or die by their tokenizers. The choice between Byte‑Pair Encoding (BPE) and SentencePiece (notably its Unigram model, but also SentencePiece‑BPE) directly shapes training efficiency, multilingual coverage, context length usage, and downstream product ergonomics. This article compares how they work, how they differ, and how to choose the right one for your model or application.

Why tokenization matters

• Compression: Fewer tokens per character/word lowers training cost and stretches context windows.
• Robustness: The ability to represent any Unicode string without unknowns (OOV) prevents brittle behavior.
• Alignment: Stable token boundaries improve attention alignment and reduce detokenization glitches.
• Throughput: Training and inference speed depend on tokenizer latency and vocabulary design.

Algorithms at a glance

BPE (Byte‑Pair Encoding)

• Core idea: Start with a base alphabet (characters or bytes) and iteratively merge the most frequent adjacent pairs to build a vocabulary of subwords.
• Variants:
  • Character‑level BPE: Starts from Unicode characters.
  • Byte‑level BPE: Starts from raw UTF‑8 bytes to guarantee zero OOV; often marks leading spaces with a special token prefix like “Ġ”.
• Deterministic encoding given a fixed merge table; no probabilistic sampling at tokenize time.

SentencePiece

SentencePiece is a library and file format that works on raw text without external pretokenization. It supports two main algorithms:

• Unigram Language Model:
  • Start from a large seed vocabulary; train a probabilistic model over pieces.
  • Iteratively prune pieces that minimally hurt likelihood; final encoding is found via Viterbi to maximize sequence probability.
  • Supports sampling during training/augmentation for robustness.
• SentencePiece‑BPE:
  • Implements BPE within SentencePiece tooling, gaining its normalization, training pipeline, and model format conveniences.
• Notable behaviors:
  • Uses the visible whitespace symbol “▁” to mark word boundaries inside tokens.
  • Provides configurable normalization (e.g., NFKC) baked into the model.

Pre‑tokenization and normalization

• BPE toolchains often rely on pretokenizers (e.g., splitting on whitespace/punctuation) plus optional byte‑level tricks. If pretokenization is language‑specific, multilingual text can fragment unpredictably.
• SentencePiece operates directly on raw text and applies consistent normalization and boundary marking. This yields stable behavior across languages without external rules.
• Normalization options matter for user‑facing fidelity: aggressive normalization (e.g., lowercasing, NFKC) can improve compression but risks altering user‑visible forms (e.g., case, compatibility characters).

Whitespace, Unicode, and OOV

• Byte‑level BPE: No OOV by construction; any byte sequence is representable. Spaces are often encoded via leading‑space markers (e.g., “ĠNew”).
• SentencePiece Unigram: Also aims for complete coverage. When trained without byte fallback, rare code points may be split into smaller pieces but remain representable. With byte fallback enabled, coverage becomes guaranteed similar to byte‑level BPE.
• Emoji and complex scripts (CJK, Thai) benefit from models trained without hardwired whitespace tokenization. SentencePiece’s raw‑text approach typically handles these more gracefully than pipelines that assume Latin‑style word boundaries.

Special tokens and offsets

• Both approaches support adding reserved tokens (e.g., BOS/EOS, padding, system/control tokens).
• Byte‑level methods simplify character‑level offsets, useful for span mapping and highlighting.
• SentencePiece’s “▁” is literal; if you display detokenized text directly, ensure you decode/strip it correctly. Libraries typically handle this automatically via the model’s decoder.

Concrete examples

Consider the string: “New York 🍕 東京”.

• Byte‑level BPE (illustrative):

  • Tokens might look like: [“ĠNew”, “ĠYork”, “Ġ”, “🍕”, “Ġ”, “東京” split into subpieces]
  • Leading spaces encoded via a marker (e.g., Ġ), emoji kept as its UTF‑8 bytes merged into a single or few tokens.

• SentencePiece Unigram (illustrative):

  • Tokens might look like: [“▁New”, “▁York”, “▁🍕”, “▁東京”] or [“▁東”, “京”] depending on training and vocab size.
  • The “▁” indicates a word boundary before each piece.

Actual outputs vary by training corpus, normalization, and vocab size.

Training recipes

Train SentencePiece (Unigram) in Python

import sentencepiece as spm

spm.SentencePieceTrainer.Train(
    input='data.txt',                # one sentence per line
    model_prefix='sp_unigram',
    vocab_size=32000,
    model_type='unigram',            # or 'bpe'
    character_coverage=0.9995,       # increase for CJK-heavy corpora
    normalization_rule_name='nmt_nfkc',
    input_sentence_size=10000000,    # subsample for speed
    shuffle_input_sentence=True,
    byte_fallback=True,              # ensures full Unicode coverage
    unk_id=0, bos_id=1, eos_id=2, pad_id=3
)

Train BPE with Hugging Face Tokenizers (byte‑level)

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import ByteLevel
from tokenizers.decoders import ByteLevel as ByteLevelDecoder

# Initialize
tokenizer = Tokenizer(BPE(unk_token="<unk>"))
tokenizer.pre_tokenizer = ByteLevel(add_prefix_space=True)
tokenizer.decoder = ByteLevelDecoder()

# Train
trainer = BpeTrainer(
    vocab_size=50000,
    min_frequency=2,
    special_tokens=["<unk>", "<s>", "</s>", "<pad>"]
)
files = ["data.txt"]
tokenizer.train(files, trainer)

# Save
tokenizer.save("bpe_bytelevel.json")

Performance: what to measure

• Compression:
  • Average tokens per byte (TPB) or per character (TPC).
  • Bits‑per‑byte (bpb) when coupled with a fixed LM; proxy by entropy or validation loss.
• Speed:
  • Tokens/sec for encode and decode on realistic batch sizes.
  • Single‑thread vs multi‑thread, CPU vs GPU offload (where supported).
• Robustness:
  • OOV rate (should be ~0 for byte‑fallback or byte‑level setups).
  • Stability under normalization and mixed‑script input.
• Memory/size:
  • Vocabulary size (e.g., 32k, 50k), merge table size (BPE), model file size (SentencePiece .model + .vocab).

A simple benchmarking harness

from time import perf_counter

def bench(tokenizer, texts):
    # warmup
    for _ in range(3):
        _ = [tokenizer.encode(t) for t in texts]
    # timed
    t0 = perf_counter()
    enc = [tokenizer.encode(t) for t in texts]
    t1 = perf_counter()
    dec = [tokenizer.decode(e.ids if hasattr(e, 'ids') else e) for e in enc]
    t2 = perf_counter()

    total_bytes = sum(len(t.encode('utf-8')) for t in texts)
    total_tokens = sum(len(e.ids if hasattr(e, 'ids') else e) for e in enc)

    return {
        'encode_ms': (t1 - t0) * 1000,
        'decode_ms': (t2 - t1) * 1000,
        'tokens_per_byte': total_tokens / total_bytes,
        'bytes_per_token': total_bytes / max(total_tokens, 1)
    }

Strengths and trade‑offs

• Byte‑level BPE

  • Pros: Guaranteed coverage; simple offsets; strong on code, URLs, emoji; widely supported and very fast.
  • Cons: Can over‑split non‑Latin scripts unless trained with rich merges; tokens with leading‑space markers may be less intuitive for humans.

• SentencePiece Unigram

  • Pros: Probabilistic model often yields compact, linguistically coherent pieces; raw‑text training avoids language‑specific pretokenizers; “▁” boundary helps word‑level reasoning; supports subword regularization.
  • Cons: Slightly more complex decoding logic; offsets can be trickier; without byte fallback, rare symbols may fragment.

• SentencePiece‑BPE

  • Pros: Combines BPE’s determinism with SentencePiece’s training and normalization pipeline; convenient model packaging.
  • Cons: Similar trade‑offs to BPE regarding multilingual fragmentation unless tuned.

Choosing for your use case

• Multilingual chat or search across mixed scripts: SentencePiece Unigram with byte_fallback and careful normalization; vocab ≈ 32k–64k.
• Code‑heavy models, log analysis, or emoji‑rich social text: Byte‑level BPE (or SP‑BPE with byte_fallback) for predictable coverage.
• Training stability on limited budgets: Start with 32k–50k vocab; evaluate TPB/TPC and adjust. Larger vocabs reduce tokens but increase softmax cost.
• Alignment‑sensitive tasks (span highlighting, RAG snippet mapping): Prefer byte‑level offsets or maintain mapping tables from tokenizer libraries.

Pitfalls and how to avoid them

• Hidden normalization drift: Ensure the same normalization at train and inference; bake rules into the model (SentencePiece) or enforce in your pipeline.
• Special token collisions: Reserve IDs consistently across checkpoints and serving stacks.
• Detokenization artifacts: Test round‑trips on user‑visible text; unit‑test punctuation, whitespace, and multi‑code‑point emojis (ZWJ sequences).
• Domain shift: Re‑train or adapt vocab on your domain corpus (code, biomedical, legal) to avoid pathological over‑segmentation.

Real‑world patterns (non‑exhaustive)

• Many general LLMs trained by various labs use byte‑level BPE or SentencePiece with byte fallback for OOV‑free robustness.
• Google‑origin models frequently adopt SentencePiece Unigram; several open LLMs use SentencePiece‑BPE with byte fallback and vocab ≈ 32k.

Production checklist

• Decide on normalization (e.g., NFKC) and stick to it.
• Guarantee full Unicode coverage (byte‑level or byte_fallback).
• Lock special‑token IDs before training.
• Benchmark TPB/TPC, encode/decode latency, and memory.
• Validate round‑trip fidelity on golden text sets spanning scripts, punctuation, and emojis.
• Version and serialize tokenizer artifacts alongside model weights.

Conclusion

There is no universally “best” tokenizer—only the best fit for your data, languages, and operational constraints. Byte‑level BPE shines for robustness and simplicity; SentencePiece Unigram offers elegant, language‑agnostic compression with flexible normalization and augmentation. With a clear benchmarking plan and disciplined normalization, either choice can deliver excellent downstream performance.