Modularcomputing/dynamic-asr

Dynamic ASR

Speech recognition for Nigerian English, Nigerian Pidgin, Igbo, Yoruba and Hausa — and for speech that switches between them inside a sentence. By Modular, part of the Dynamic Accent project (see also the Dynamic Accent TTS and the Dynamic Accent Duplex voice agent, which uses this model as its ASR).

Dynamic ASR is Meta's Omnilingual ASR omniASR_CTC_1B_v2 (1B-parameter wav2vec2 CTC, Apache-2.0) fully fine-tuned on 187.2 hours of licensed Nigerian speech. The architecture, tokenizer and decoding are unchanged, so it is a drop-in replacement: language-agnostic (no language hint), greedy CTC, cheap enough to re-decode a growing buffer for streaming partials.

Status: experimental research release. Read the limitations.

Two models

folder	what it is	use it when
real/ (default)	fine-tuned on licensed real speech only	general use; as good as the experimental model on real code-switched speech and better on the selection dev set (mean CER 0.071 vs 0.074)
codeswitch-experimental/	same recipe, plus ~4.99 h of synthetic code-switched speech made with our own Dynamic Accent Flash-DPO TTS (12 % of training audio)	research on synthetic code-switch data; see the ablation below before choosing it

Both are the same size (1B, ~1.95 GB bf16 state dict) and load the same way.

Variants

folder	model	parameters	files	runtime
real/ (default)	omniASR_CTC_1B_v2 fine-tuned on licensed real speech	1B	model.pt (1.95 GB, bf16)	PyTorch / fairseq2 (transcribe.py)
codeswitch-experimental/	same, plus synthetic code-switched speech	1B	model.pt (1.95 GB, bf16)	PyTorch / fairseq2 (transcribe.py)
pocket/	omniASR_CTC_300M_v2 student distilled from a 1B Dynamic ASR teacher	300M	model_int8.onnx (367 MB), model_fp32.onnx (1.30 GB), model.pt (1.30 GB), vocab.json	onnxruntime on CPU (pocket/transcribe_pocket.py)

The Pocket numbers below use a different scorer from the 1B tables in Results, so compare Pocket only with the teacher row in its own table.

Pocket (300M, CPU)

pocket/ is a smaller model for CPU and offline use. It is Meta's omniASR_CTC_300M_v2 (300M-parameter wav2vec2 CTC, Apache-2.0, same tokenizer and 20 ms frame rate as the 1B models), distilled from a 1B Dynamic ASR teacher. The teacher is an internal checkpoint (real/ further fine-tuned with real-voice spliced code-switch clips); it is not one of the files in this repository. Checkpoint: P2, step 2500.

Training. Loss = CTC on the reference transcripts + KL divergence to the teacher's per-frame distribution; CNN feature encoder frozen. Data: the licensed real speech listed under Training data, code-switch clips spliced from that same licensed speech, and an oversample of 0.9 h of real code-switched training clips. No synthetic TTS audio and no African Voices data.

Files

file	size	what it is
pocket/model_int8.onnx	367 MB	int8-quantised ONNX (recommended for CPU)
pocket/model_fp32.onnx	1.30 GB	fp32 ONNX
pocket/model.pt	1.30 GB	PyTorch checkpoint: a dict with model (state dict for omniASR_CTC_300M_v2), step and the training args
pocket/vocab.json	127 KB	CTC vocabulary (index 0 = blank, 1–3 = special tokens)
pocket/transcribe_pocket.py		CPU inference: onnxruntime, greedy CTC decoding, 16 kHz mono input
pocket/results.json		the scores and CPU benchmark below

ONNX graph: input wav (float32, [1, samples], 16 kHz mono), output logprobs ([1, frames, 10288]).

Usage

pip install onnxruntime numpy soundfile scipy huggingface_hub
huggingface-cli download Modularcomputing/dynamic-asr --include "pocket/model_int8.onnx" "pocket/vocab.json" "pocket/transcribe_pocket.py" --local-dir .
python pocket/transcribe_pocket.py audio.wav                                  # int8 (default)
python pocket/transcribe_pocket.py audio.wav --model pocket/model_fp32.onnx   # fp32

Any wav is mixed to mono and resampled to 16 kHz; clips over ~20 s are cut at quiet points and transcribed in pieces. Output format is the same as the 1B models.

Results (CER, greedy CTC, no language model). gdev = the general dev set (WAXAL validation + OpenSLR-70, 187 clips); it was used for checkpoint selection, so it is a dev set, not a test set. Real code-switch = the held-out 94-clip real code-switched test set described above, never used for training or selection. Scoring: NFC, lowercase, punctuation and apostrophes removed, English number words to digits, Igbo tone marks removed from both sides, Yoruba tone marks kept.

model	English	Hausa	Igbo	Pidgin	Yoruba	gdev mean	real code-switch
Pocket int8 ONNX	0.0571	0.0522	0.0595	0.0963	0.1359	0.0802	0.1558
Pocket fp32 ONNX	0.0571	0.0514	0.0595	0.0925	0.1350	0.0791	0.1558
1B teacher (greedy)	0.0294	0.0489	0.0598	0.0823	0.1288	0.0698	0.1375

• int8 adds about 0.001 to the gdev mean (mostly Pidgin) and nothing on real code-switch.
• English is the largest gap to the teacher (0.057 vs 0.029).

CPU speed (Intel Xeon Platinum 8480+, batch 1, 20 clips totalling 110 s, one fresh process per setting)

model	threads	RTF	median latency per clip	peak RAM
int8 ONNX	4	0.071	358 ms	939 MB
fp32 ONNX	4	0.186	952 ms	2287 MB

Pocket shares the limitations listed below, with higher error than the 1B models.

Samples

Hear the input and read what the model wrote. The input clips are synthetic speech from our Dynamic Accent Flash-DPO TTS (female waxal_ibo_5 or male anchor voice), not real recordings, so they are cleaner than real-world audio. The transcripts come from an internal Dynamic ASR checkpoint newer than real/ (it adds a Yoruba tone-mark fix and is not one of the files in this repository yet): with LM = decoding with a 4-gram KenLM, as the Duplex agent uses on final segments; greedy = plain CTC, no LM. Output is lower-case with no punctuation, and Igbo has no tone marks by design. Audio: mono MP3, CC-BY-SA-4.0.

1. Igbo (female TTS voice)

	text
reference	Ndewo, kedụ ka ị mere? Anyị ga-aga ahịa echi n'ụtụtụ.
Dynamic ASR, with LM	nde wo kedu ka i mere anyị ga agaghi echi n'ụtụtụ
Dynamic ASR, greedy	ndewo kedu ka ị mere anyị ga aga ahị echi n'ụtụtụ

2. Yoruba (female TTS voice)

	text
reference	Ẹ káàárọ̀! Ṣé dáadáa ni ẹ wà? Mo fẹ́ lọ sí ọjà lọ́la.
Dynamic ASR, with LM	ẹkáàárọ̀ ṣé dáadáa ni ẹwà mo fẹ́ lọ sí ọjà lọ́lá
Dynamic ASR, greedy	ẹká àrọ̀ ṣé dáadá ni ẹwà mo fẹ́ lọ sí ọjà lọla

3. Hausa (male TTS voice)

	text
reference	Sannu da zuwa. Yaya aiki? Zan tafi Kano gobe da safe.
Dynamic ASR, with LM	sannu da zuwa yaya aiki zan tafi kano gobe a safe
Dynamic ASR, greedy	sanuda zuwa yaya aiki zan tafi kano gube da safe

4. Pidgin (female TTS voice)

	text
reference	How far, my guy? Abeg make we see for evening, I dey come.
Dynamic ASR, with LM	how far my guy abeg make we see for evenin i dey come
Dynamic ASR, greedy	how far my guy abeg make we see for evening i dey come

5. Igbo + English (code-switch) (male TTS voice)

	text
reference	Biko, can you send me the report before Friday? Daalụ.
Dynamic ASR, with LM	biko can you send me the report before friday talo
Dynamic ASR, greedy	biko can you send me the report before friday daalo

Results

All test data is held out (never trained on). Scores are WER / CER after lower-casing and stripping punctuation; "no-diacritic" CER folds tone marks and dots. Decoding: greedy CTC with a Latin-script logit mask (the duplex setting).

Per language (duplex evaluation set: 229 clips — WAXAL test ibo/yor/hau/pcm × 40, FLEURS validation English × 29, 40 clips made by joining two held-out clips of different languages)

CER

model	English	Pidgin	Igbo	Yoruba	Hausa	code-switch (concatenated)
omniASR_CTC_1B (stock; previous duplex ASR)	0.065	0.169	0.206	0.191	0.090	0.128
omniASR_CTC_1B_v2 (stock base)	0.024	0.123	0.133	0.135	0.106	0.104
Dynamic ASR real (real/)	0.025	0.074	0.093	0.121	0.082	0.081
Dynamic ASR code-switch, experimental (codeswitch-experimental/)	0.024	0.076	0.095	0.118	0.081	0.072

WER

model	English	Pidgin	Igbo	Yoruba	Hausa	code-switch (concatenated)
omniASR_CTC_1B (stock; previous duplex ASR)	0.144	0.380	0.636	0.498	0.299	0.330
omniASR_CTC_1B_v2 (stock base)	0.097	0.336	0.456	0.401	0.317	0.292
Dynamic ASR real (real/)	0.095	0.193	0.315	0.355	0.253	0.238
Dynamic ASR code-switch, experimental (codeswitch-experimental/)	0.092	0.193	0.321	0.353	0.252	0.216

Real code-switched speech

Licensed corpora contain little real code-switched speech, so we built the most honest test we could from held-out data: Igbo/Yoruba/Hausa utterances whose transcripts contain English words. Natural conversational = spontaneous Igbo speech with English (IgboSynCorp; those speakers were removed from training). Insertional = read Igbo/Yoruba/Hausa sentences with English names/terms (WAXAL test, NaijaS2ST dev). CER (no-diacritic CER in brackets), plus English-word recall (share of the reference's English words that appear in the output; strict exact-match, so low in absolute terms). The last column is a held-out set of synthetic code-switched clips (unseen texts) from the TTS.

model	real CS, all (n=94)	natural conversational Igbo-English (n=36)	insertional, read (n=58)	English-word recall	synthetic CS test (n=109)
omniASR_CTC_1B (stock; previous duplex ASR)	0.224 (0.195)	0.372 (0.326)	0.178 (0.154)	0.3009	0.159 (0.129)
omniASR_CTC_1B_v2 (stock base)	0.193 (0.172)	0.353 (0.315)	0.143 (0.128)	0.2832	0.132 (0.118)
Dynamic ASR real (real/)	0.148 (0.137)	0.248 (0.234)	0.117 (0.107)	0.3186	0.105 (0.091)
Dynamic ASR code-switch, experimental (codeswitch-experimental/)	0.149 (0.137)	0.247 (0.234)	0.119 (0.106)	0.3805	0.038 (0.035)

Did synthetic code-switched audio help?

Short answer: not on real code-switched speech — at least not measurably at this scale. The two models share everything (data, seed, 4,000 steps) except that codeswitch-experimental/ sampled 12 % of its audio from 2572 synthetic code-switched clips (4.99 h).

• Real code-switched speech (94 clips): a tie. CER 0.148 (real) vs 0.149 (codeswitch-experimental); natural conversational Igbo–English 0.248 vs 0.247; insertional 0.117 vs 0.119; WER 0.436 vs 0.438. All differences are well inside the noise of a 94-clip test.
• English-word recall did go up, 0.319 → 0.381 (share of the reference's English words the model writes out exactly). This is the one real-speech signal in favour of synthetic data: the model spells embedded English words as English more often. It did not translate into lower CER, and the test is small, so treat it as a lead, not a result.
• Per-language real tests: no change. Differences of at most 0.003 CER on English, Pidgin, Igbo, Yoruba and Hausa (B slightly better on Yoruba/Hausa/English, slightly worse on Pidgin/Igbo). On the selection dev set (WAXAL validation + OpenSLR-70) the experimental model is slightly worse (mean CER 0.074 vs 0.071).
• Concatenated code-switch clips (real audio of two languages joined): small gain, CER 0.081 → 0.072, WER 0.238 → 0.216. These are inter-sentential switches at a hard boundary, closer to what the synthetic data looks like than to natural speech.
• Synthetic code-switched test (109 held-out clips, unseen texts): large gain, CER 0.105 → 0.038 (stock v2 base: 0.132). This mostly measures adaptation to our own TTS voices and recording style, so it is not evidence about real speakers.

Verdict: synthetic code-switched audio from our TTS teaches the model the synthetic domain and slightly improves English-word spelling, but it does not lower error on real code-switched or monolingual speech. real/ stays the default. A real natural code-switching test set (and more real code-switched training speech) is what would settle this.

AfriSwitch (in-the-wild code-switched speech) — pending

We planned to score both models (and the stock v2 base) zero-shot on the Nigerian subsets of Intron Health's AfriSwitch benchmark (Igbo, Yoruba, Hausa, Nigerian Pidgin; evaluation only, never trained on). The dataset is gated with manual approval; we requested access on 2026-10-24 and it had not been granted when the models were finalised, so no AfriSwitch scores are reported yet. When we run it we will use our scorer's normalisation (lowercase, punctuation stripped; the paper scores diacritic-preserving WER, and our models drop Igbo tone marks, so Igbo WER will be inflated against its references and we will report the diacritic-folded figure alongside), corpus-level WER and CER.

For reference, the paper's zero-shot WER (%) on these subsets (Table 2; systems given the matrix language, diacritics preserved):

language	Sahara V2.5 (best overall, avg 35.93 over 12 languages)	Omnilingual LLM 7B	Gemini 3.6	ElevenLabs	Sahara V2
Igbo	46.94	75.52	75.88	65.37	83.58
Yoruba	45.54	70.27	77.26	74.62	89.66
Hausa	31.77	39.27	45.44	41.01	45.56
Nigerian Pidgin	32.83	42.71	38.84	35.33	42.19

AfriSwitch: Ashungafac, Awobade & Olatunji (Intron Health), AfriSwitch: A Benchmark for In-the-Wild African Code-Switched Speech Recognition, arXiv:2608.26434 (2026). Dataset: https://huggingface.co/datasets/intronhealth/AfriSwitch (dataset card licence: CC-BY-NC-SA-4.0; paper CC BY 4.0).

@misc{ashungafac2026afriswitch,
  title={AfriSwitch: A Benchmark for In-the-Wild African Code-Switched Speech Recognition},
  author={Ashungafac, Gabrial Zencha and Awobade, Busayo and Olatunji, Tobi},
  year={2026}, eprint={2608.26434}, archivePrefix={arXiv}, primaryClass={cs.CL}
}

Usage

pip install "omnilingual-asr>=0.2.0" huggingface_hub soundfile scipy
python transcribe.py audio.wav                      # real/ model (default)
python transcribe.py audio.wav --model codeswitch   # codeswitch-experimental/

from transcribe import DynamicASR
asr = DynamicASR("real")          # downloads real/model.pt; base architecture omniASR_CTC_1B_v2
print(asr(waveform, sr))          # mono float32 array, any sample rate

The checkpoint is a plain PyTorch state dict ({"model": state_dict, ...}) for omniASR_CTC_1B_v2: transcribe.py loads the base through fairseq2.models.hub.load_model("omniASR_CTC_1B_v2") and overwrites its weights (the base download is needed once for the model skeleton and tokenizer). On omnilingual-asr 0.1.0, run code/training/patch_v2.sh (adjust the venv path) to register the v2 card first.

Streaming (duplex agent). code/asr_server.py is the FastAPI worker used by Dynamic Accent Duplex: POST /asr with raw float32 16 kHz mono → {"text", "ms"}; partial hypotheses come from re-decoding the growing buffer every 0.64 s (~30–90 ms per call on an H100).

cd code && ASR_MODEL=dynamic-asr uvicorn asr_server:app --port 8002      # real/ (default)
#          ASR_MODEL=dynamic-asr-cs  -> codeswitch-experimental/
#          ASR_MODEL=omni            -> stock omniASR_CTC_1B (also the automatic fallback if Dynamic ASR cannot load)

Output format: lower-case, no punctuation, digits as spoken or written in the training transcripts. Igbo is written without tone marks (dotted letters ị ọ ụ ṅ kept, standard everyday orthography); Yoruba with tone marks; Hausa as in the sources (hooked letters often missing, as in WAXAL).

Training data (licensed only)

source	language	hours used	licence
bible_hau	hau	27.01	CC-BY-SA-4.0 (Open.Bible / Biblica)
bible_yor	yor	20.43	CC-BY-SA-4.0 (Open.Bible / Biblica)
fleurs	eng	5.63	CC-BY-4.0 (Google FLEURS)
fleurs	hau	7.6	CC-BY-4.0 (Google FLEURS)
fleurs	ibo	6.09	CC-BY-4.0 (Google FLEURS)
fleurs	yor	5.74	CC-BY-4.0 (Google FLEURS)
igboapi	ibo	34.98	CC-BY-4.0 (IgboAPI)
igbosyncorp	ibo	2.16	CC0-1.0 (IgboSynCorp)
naijas2st_igbo	ibo	14.81	CC-BY-4.0 (NaijaS2ST)
open_bible_igbo	ibo	22.02	CC-BY-SA-4.0 (Open.Bible / Biblica)
slr70	eng	4.0	CC-BY-SA-4.0 (OpenSLR 70, Nigerian English)
slr86	yor	2.98	CC-BY-SA-4.0 (OpenSLR 86, Yoruba)
waxal	hau	6.58	CC-BY-SA-4.0 (WAXAL Media Trust subsets)
waxal	ibo	10.58	CC-BY-SA-4.0 (WAXAL Media Trust subsets)
waxal	pcm	6.07	CC-BY-SA-4.0 (WAXAL Media Trust subsets)
waxal	yor	10.48	CC-BY-SA-4.0 (WAXAL Media Trust subsets)

Per-epoch sampling down-weights Bible narration and dictionary reads and up-weights Pidgin (≈ 115.6 h per epoch; Igbo 39, Yoruba 26, Hausa 24, Pidgin 18, English 8 h). Held out and never trained on: WAXAL test + validation, FLEURS dev/test, NaijaS2ST dev, and every IgboSynCorp speaker whose speech contained English.

Synthetic code-switched data (codeswitch-experimental/ only). 2797 code-switched sentences (English with Igbo/Yoruba/Hausa/Pidgin, Pidgin with Igbo/Yoruba/Hausa; spans tagged per language) written by Qwen3-30B-A3B-Instruct-2507 (Apache-2.0), read by the Dynamic Accent Flash-DPO TTS (itself trained on licensed data only) in 32 voices (zero-shot prompts from WAXAL speakers), with random speaking rate (slow/normal/fast), speed perturbation 0.9–1.1, and on-the-fly reverb, phone/cheap-mic band-limiting, clipping and noise (SNR 5–30 dB). Clips the stock model could not read (CER > 0.35, i.e. TTS failures) were dropped: 2572 clips / 4.99 h kept for training, 109 unseen-text clips kept as the synthetic test.

Not used, on licence grounds: African Voices / WaZoBia, NaijaVoices (non-commercial), AfriSpeech-200 (non-commercial). Common Voice was not used (not fetched). No African Voices–trained TTS was used to make synthetic data.

Training

Full fine-tune of all transformer layers (CNN feature encoder frozen), bf16 autocast, AdamW (lr 2e-5, β 0.9/0.98, wd 0.01), 300 warm-up steps then linear decay, 4000 steps × 160 s of audio per batch (~177.8 h of audio seen), CTC loss, LayerDrop 0.1, 15 % light augmentation of real clips. One H100; the two models differ only in the synthetic share (0 % vs 12 %), same seed and steps. Checkpoint selected on a WAXAL-validation/OpenSLR-70 dev set. Training, data-prep, synthesis and evaluation code is in code/training/.

Limitations

• Small, read-speech-heavy test sets. 40 clips per language; real code-switched test = 94 clips. Differences of ~0.01 CER are within noise.
• Convention matching is part of the gain. WAXAL train and test share transcription conventions (e.g. Igbo without tones, Hausa without hooked letters, digits as numerals); some of the improvement on WAXAL tests is learning those conventions, not only better hearing.
• Little natural conversational speech (≈2 h IgboSynCorp in training; none for Yoruba/Hausa). Expect higher error rates on spontaneous, noisy, phone or far-field speech, and on accents unlike the training speakers.
• Code-switching is the weakest area. Real intra-sentential code-switched test data is scarce; the "natural conversational" CER is still ~0.25. English inside Igbo/Yoruba/Hausa is often spelt phonetically.
• Yoruba tone marks are frequently wrong or missing (Yoruba CER without diacritics is much lower than with); Igbo output has no tone marks by design.
• Not evaluated yet on the developer's own voice or other unseen real users; not evaluated for fairness across gender/age/region.
• No punctuation or casing (unlike the stock v2 model's written output).
• Not for high-stakes use (medical, legal, emergency) without human review.

Licence and attribution

• Weights: CC-BY-SA-4.0. Several training sources are CC-BY-SA-4.0 (WAXAL Media Trust subsets, Open.Bible, OpenSLR 70/86), so the fine-tuned weights are shared alike under CC-BY-SA-4.0. The base model omniASR_CTC_1B_v2 is Apache-2.0 (© Meta Platforms).
• Code (transcribe.py, code/, pocket/transcribe_pocket.py): Apache-2.0 (LICENSE-CODE).
• Pocket weights (pocket/): CC-BY-SA-4.0, like the 1B weights. Base model omniASR_CTC_300M_v2 is Apache-2.0 (© Meta Platforms).
• Attribution: Google WAXAL (Media Trust subsets), Google FLEURS (CC-BY-4.0), OpenSLR 70 Nigerian English and OpenSLR 86 Yoruba (Google, CC-BY-SA-4.0), Open.Bible / Biblica (CC-BY-SA-4.0), NaijaS2ST (CC-BY-4.0), IgboAPI (CC-BY-4.0), IgboSynCorp (CC0-1.0, Harvard Dataverse doi:10.7910/DVN/YB9FWK), Meta Omnilingual ASR (Apache-2.0), Qwen3 (Apache-2.0).

Citation

@misc{modular2026dynamicasr,
  title  = {Dynamic ASR: Nigerian multilingual and code-switched speech recognition},
  author = {Modular (Chima Emmanuel Akuegbo)},
  year   = {2026},
  url    = {https://huggingface.co/Modularcomputing/dynamic-asr}
}