Somali Tokenizer

Somali ByteLevel BPE v2 (formerly somast/somali-bpe-v2, which still redirects here).

A 48,000-token byte-level BPE tokenizer for Somali, trained by Somast for the Somali NLP Corpus.

  • Byte-level, so every input round-trips exactly and there is no unknown token.
  • 48,000 types, including special tokens <|endoftext|>, <|pad|>, <|im_start|>, <|im_end|> and 12 reserved slots for a decoder-LM control set.
  • Context length set to 8,192 in tokenizer_config.json.

Use

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("somast/somali-tokenizer")
ids = tok.encode("Caafimaadka carruurtu waa mudnaanteenna koowaad.")
print(tok.convert_ids_to_tokens(ids))
# ['Caafimaadka', 'Ġcarruurtu', 'Ġwaa', 'Ġmudnaan', 'teenna', 'Ġkoowaad', '.']
print(tok.decode(ids))  # the input, exactly

Or with the somali-tokenizer package, a Rust implementation with Python bindings that validates the artifact on load and gives the same ids:

# pip install somali-tokenizer
from somali_tokenizer import SomaliTokenizer

tok = SomaliTokenizer.from_pretrained("somast/somali-tokenizer")  # or SomaliTokenizer.v2() offline
enc = tok.encode("Caafimaadka carruurtu waa mudnaanteenna koowaad.")
print(enc.tokens)
print(tok.decode(enc.ids))

The package also ships a command-line tool (somali-tokenizer encode, train, benchmark, stats) and a Rust crate of the same name.

To pin this exact artifact, pass revision="v1.0.0".

Compared with general-purpose tokenizers

FLORES-200 Somali (dev + devtest, 2,009 sentences, 45,375 words). Tokens per word, lower is better; no special tokens counted.

Tokenizer Vocabulary Tokens per word Single-token words Tokens vs this
somast/somali-tokenizer (this) 48,000 1.374 75.2% 1.00×
NLLB-200 256,204 1.818 50.8% 1.32×
XLM-RoBERTa 250,002 1.827 50.9% 1.33×
GPT-4o (o200k_base) 200,019 1.975 45.3% 1.44×
Gemma 2 256,000 2.241 35.2% 1.63×
Llama 3 128,256 2.527 23.4% 1.84×
GPT-4 (cl100k_base) 100,277 2.552 22.9% 1.86×
Qwen 2.5 151,665 2.568 22.9% 1.87×

On 2,000 Somali Wikipedia paragraphs the order is the same: 1.505 tokens per word here, 2.060 for GPT-4o, 2.632 for GPT-4. The example sentence above is 7 tokens here, 16 with GPT-4o and 19 with GPT-4 and Llama 3. Full tables, method, and caveats (Wikipedia, and possibly FLORES, may overlap the training corpus) are in the benchmark report and BENCHMARKING.md.

Results on the corpus holdout

Scored on a held-out split of 63,542 documents (about 0.8% of the Somali NLP Corpus 1.0.0, excluded by a deterministic hash of the document id). Mean tokens per word, lower is better:

Tokenizer Tokens per word Round-trip
Somali ByteLevel BPE v2 (this) 1.3440 exact on all 63,542
Somali BPE v1 (32k, whitespace) 1.3830 fails on every document
XLM-RoBERTa-base 1.8100 n/a
BERT-base-uncased 2.6344 n/a

Over the whole 1.0.0 corpus (7,913,008 documents, 826,973,689 words) this tokenizer produces 1,129,084,663 tokens, 1.3653 per word. Per-source figures and the full method are in the Somali-Tokenizer repository, including the technical note (python-reference/PAPER.md).

Limits

  • The vocabulary was trained on an earlier eleven-source build of the corpus (2026-09-01) and was not retrained for the six sources added afterwards (Glot500, Somali Web Corpus, FinePDFs, FineWeb-2, Somali Alpaca, Somali TinyStories). FinePDFs tokenizes above the holdout mean (1.4905), so a retrained vocabulary should help on PDF-style text.
  • Tuned for Somali. Other languages work, because the tokenizer is byte-level, but are split inefficiently.
  • Only the tokenizer is licensed here. The corpus it was trained on is released separately, under the per-record licenses described on its card.

Citation

@misc{somali_tokenizer,
  title  = {Somali Tokenizer: Somali ByteLevel BPE v2},
  author = {omartood and sharafdin and Somast},
  year   = {2026},
  url    = {https://huggingface.co/somast/somali-tokenizer}
}

Contact: info@somast.so

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support