Somali Tokenizer
Somali ByteLevel BPE v2 (formerly somast/somali-bpe-v2, which still redirects here).
A 48,000-token byte-level BPE tokenizer for Somali, trained by Somast for the Somali NLP Corpus.
- Byte-level, so every input round-trips exactly and there is no unknown token.
- 48,000 types, including special tokens
<|endoftext|>,<|pad|>,<|im_start|>,<|im_end|>and 12 reserved slots for a decoder-LM control set. - Context length set to 8,192 in
tokenizer_config.json.
Use
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("somast/somali-tokenizer")
ids = tok.encode("Caafimaadka carruurtu waa mudnaanteenna koowaad.")
print(tok.convert_ids_to_tokens(ids))
# ['Caafimaadka', 'Ġcarruurtu', 'Ġwaa', 'Ġmudnaan', 'teenna', 'Ġkoowaad', '.']
print(tok.decode(ids)) # the input, exactly
Or with the somali-tokenizer package, a
Rust implementation with Python bindings that validates the artifact on load and gives the
same ids:
# pip install somali-tokenizer
from somali_tokenizer import SomaliTokenizer
tok = SomaliTokenizer.from_pretrained("somast/somali-tokenizer") # or SomaliTokenizer.v2() offline
enc = tok.encode("Caafimaadka carruurtu waa mudnaanteenna koowaad.")
print(enc.tokens)
print(tok.decode(enc.ids))
The package also ships a command-line tool (somali-tokenizer encode, train,
benchmark, stats) and a Rust crate of the same name.
To pin this exact artifact, pass revision="v1.0.0".
Compared with general-purpose tokenizers
FLORES-200 Somali (dev + devtest, 2,009 sentences, 45,375 words). Tokens per word, lower is better; no special tokens counted.
| Tokenizer | Vocabulary | Tokens per word | Single-token words | Tokens vs this |
|---|---|---|---|---|
| somast/somali-tokenizer (this) | 48,000 | 1.374 | 75.2% | 1.00× |
| NLLB-200 | 256,204 | 1.818 | 50.8% | 1.32× |
| XLM-RoBERTa | 250,002 | 1.827 | 50.9% | 1.33× |
GPT-4o (o200k_base) |
200,019 | 1.975 | 45.3% | 1.44× |
| Gemma 2 | 256,000 | 2.241 | 35.2% | 1.63× |
| Llama 3 | 128,256 | 2.527 | 23.4% | 1.84× |
GPT-4 (cl100k_base) |
100,277 | 2.552 | 22.9% | 1.86× |
| Qwen 2.5 | 151,665 | 2.568 | 22.9% | 1.87× |
On 2,000 Somali Wikipedia paragraphs the order is the same: 1.505 tokens per word here, 2.060 for GPT-4o, 2.632 for GPT-4. The example sentence above is 7 tokens here, 16 with GPT-4o and 19 with GPT-4 and Llama 3. Full tables, method, and caveats (Wikipedia, and possibly FLORES, may overlap the training corpus) are in the benchmark report and BENCHMARKING.md.
Results on the corpus holdout
Scored on a held-out split of 63,542 documents (about 0.8% of the Somali NLP Corpus 1.0.0, excluded by a deterministic hash of the document id). Mean tokens per word, lower is better:
| Tokenizer | Tokens per word | Round-trip |
|---|---|---|
| Somali ByteLevel BPE v2 (this) | 1.3440 | exact on all 63,542 |
| Somali BPE v1 (32k, whitespace) | 1.3830 | fails on every document |
| XLM-RoBERTa-base | 1.8100 | n/a |
| BERT-base-uncased | 2.6344 | n/a |
Over the whole 1.0.0 corpus (7,913,008 documents, 826,973,689 words) this tokenizer produces
1,129,084,663 tokens, 1.3653 per word. Per-source figures and the full method are in the
Somali-Tokenizer repository, including the
technical note (python-reference/PAPER.md).
Limits
- The vocabulary was trained on an earlier eleven-source build of the corpus (2026-09-01) and was not retrained for the six sources added afterwards (Glot500, Somali Web Corpus, FinePDFs, FineWeb-2, Somali Alpaca, Somali TinyStories). FinePDFs tokenizes above the holdout mean (1.4905), so a retrained vocabulary should help on PDF-style text.
- Tuned for Somali. Other languages work, because the tokenizer is byte-level, but are split inefficiently.
- Only the tokenizer is licensed here. The corpus it was trained on is released separately, under the per-record licenses described on its card.
Citation
@misc{somali_tokenizer,
title = {Somali Tokenizer: Somali ByteLevel BPE v2},
author = {omartood and sharafdin and Somast},
year = {2026},
url = {https://huggingface.co/somast/somali-tokenizer}
}
Contact: info@somast.so