109M · 149MBase · InstructApache 2.0

Models. Kiswahili, from the tokenizer up.

KW5 is our family of Kiswahili language models. The tokenizer and the weights were trained from scratch, and every release is published under Apache 2.0 with its evaluations and its limits.

Weights on Hugging Face Technical dossier

Releases: 5
Sizes: 109M · 149M
Pretraining: 1.41B · 1.97B tokens
Licence: Apache 2.0

KW5-149M. The second generation.

Twenty layers with Canon layers and grouped-query attention, pretrained on 1.97B tokens of Kiswahili on a Kaggle TPU v5e-8 in about 27 TPU-hours. It is the base the instruction releases are built on.

KW5-149M in full

Stored parameters: 173.6M; 149.0M by release name; embeddings untied
Layers: 20; 768 wide, feed-forward 2,048
Context: 2,048; tokens
Vocabulary: 32,000; BPE, byte fallback

KW5-149M · one token's path

Tokenizer
Embedding
RMSNorm
Attention
Canon layers
SwiGLU
Document mask
×20 blocks
Output head

Nine stages, from text to the next token.

Chosen in a bake-off

Before the full run, the architecture was chosen against a control: with Canon layers and the Muon optimizer, a proxy model reached a validation loss of 3.8129 against 3.9829 for the first generation's design.

Trained on eight registers

Web
News
Encyclopedia
Translated educational
Parliamentary
Religious
Long-form
Chat format

1.97B

It saw 75.2% of one pass over a 2.62B-token Kiswahili corpus, with the learning rate decayed to the end.

What we measured.

Nine models were run on the same Kiswahili tests in September; seven completed. KW5-149M has the lowest bits per byte, 0.926, and 73.1% on MasakhaNEWS topic classification, level with XGLM-564M. The news text is in its training domain, and the card says so.

Every result

Bits per byte on MasakhaNEWS test text, a domain KW5 was trained on. Lower is better.

Bits per byte by model, lowest first
Model	Bits per byte
KW5-149M	0.926
KW5-149M-instruct	0.933
XGLM-564M	1.122
BLOOM-560M	1.699
GPT-2 124M	2.126
Pythia-410M	2.285
Qwen2.5-0.5B	2.393

120 documents, 320,701 bytes, seed 20260914, run 14 September 2026. Goldfish swa 100MB and Gemma 3 270M did not load and stay in the record.

The evidence

Belebele and AfriXNLI did not separate any model from chance in this run, so they are not cited here as wins.

KW5-149M-instruct-r2. Answers from the passage, declines the rest.

The revised instruction release, fully fine-tuned on KW5-149M. On a frozen suite of 924 items it reads and answers better than the releases before it, and it is better at saying it does not know.

KW5-149M-instruct-r2 in full

43.2%

Questions over real Kiswahili passages, answered from the passage (n = 250, 95% interval 37.2 to 49.4).

55.0%

Questions the passage cannot answer, declined rather than invented (n = 40). The first instruct release declined 42.5%.

The rest of the family.

Kept public on purpose: each is the baseline the next was measured against.

Second generation · instruct

KW5-149M-instruct

The first fine-tune of the second generation: two epochs over 465M tokens, about a tenth of the mix teaching the model to decline. Its repetition in follow-up turns is documented in the audit; r2 is the revision.

The release

First generation · base

KW5-109M

A Llama-compatible decoder, twelve layers, trained on one T4 GPU in about 60 hours over 1.41B tokens: two passes over the 705M unique tokens of cleaned Kiswahili web text then available. The corpus was smaller than planned, so its learning rate never decayed. The second generation began from that lesson.

The release

First generation · instruct

KW5-109M-instruct

A LoRA fine-tune, merged into the weights, over about 107,000 examples, 137 of them teaching abstention. The baseline every later instruction release is measured against.

The release

Failures stay in the record.

Moving training to the TPU broke the tie between the input embedding and the output head. The model trained correctly, untied; every reload then rebuilt it tied and overwrote the embedding. It was believed lost for a month. The fix is guarded by tests, and the postmortem is published with the cards.

LimitConfident errorsIt states wrong facts fluently. Do not use it as a source of facts.
LimitNo arithmeticBare arithmetic and word problems are near zero for every release.
LimitNo safety tuningIt has had no real safety training. It is for research and fine-tuning.
LimitMostly TanzanianPrimarily Tanzanian Kiswahili, partly from machine-translated text, not yet reviewed by native speakers.

Where it runs, and where it does not yet.

696 → 251 MB

With weight-only int8, KW5-149M shrinks to 251 MB and runs on a laptop CPU; bits per byte moves by 0.0001 at most. Our local chat app runs all five checkpoints on a CPU.

Not in a product yet

No Regnant system ships KW5 today. The systems run open models through Cordon and Orbis uses general models for Kiswahili lessons. The second generation needs its own modelling code, so it is not yet a drop-in for llama.cpp.