Models · Third generation · Base · Current
KW5-230M The third-generation base.
The third-generation base: 230M parameters, pretrained from scratch on 7.90B tokens of Kiswahili.
A base model continues text; it does not follow instructions. It is the foundation of KW5-230M-instruct, and the starting point for anyone fine-tuning a small Kiswahili model on their own data.
- 230.4M
- Parameters
- embeddings tied
- 24
- Layers
- 896 wide, GQA 14:2
- 7.90B
- Tokens
- 34.3 per stored parameter
- Apache 2.0
- Licence
How it was trained.
- Corpus
- 5.58B tokens: the audited v2 corpus, translated educational web, long-form PDFs, Swahili FineWiki, generated arithmetic and dialogue
- Tokens seen
- 7,895,777,280 over 15,060 steps of 524,288, about 1.4 passes
- Architecture
- Pre-norm decoder: RMSNorm with QK-norm, SwiGLU, RoPE, grouped-query attention (14 query, 2 key-value heads), Canon layers, document-boundary masking, tied embeddings
- Schedule
- Warmup-stable-decay; context 1,024, then 2,048 for the decay over the last 15%
- Optimiser
- Muon for matrices, AdamW for the tied embedding and the norms
- Hardware
- Kaggle TPU v5e-8, five sessions
- Validation loss
- 2.3656 at step 15,000
How it measured.
- Bits per byte, 120 MasakhaNEWS test articles
- 0.891 (KW5-149M 0.926, Goldfish 125M 0.894, XGLM-564M 1.122)
- MasakhaNEWS-sw topic accuracy, n = 476
- 74.2% [70.0, 77.9]
- Belebele-sw, n = 900
- 33.0% [30.0, 36.1], chance 25%
- AfriXNLI-sw, n = 600
- 36.5%, p = 0.055 against chance
News is in the pretraining mix and the test articles were not checked for overlap, so bits per byte and MasakhaNEWS are in-domain. Every model compared beats chance on Belebele; none clears it on AfriXNLI.
What it does well.
- Fluent Kiswahili continuation
- Arithmetic with its working in the pretraining layout: 56 of 57 bare sums, 56 of 60 with brackets
- Clears Chinchilla's 20 tokens per parameter with room
- Encodes line by line, so line-initial words keep their word-boundary piece
What it gets wrong.
- Not an instruction model
- States facts confidently and wrongly
- Loops on open prompts without a repetition penalty
- Primarily Tanzanian Kiswahili; part of the corpus is machine-translated
- No safety tuning
The family.
Run it on your own hardware.
The model card has the exact prompt format and the inference code.