KW5-149M. The second-generation base.
The second-generation base model: 149M parameters, pretrained from scratch on 1.97B tokens of Kiswahili.
A base model continues text; it does not follow instructions. It is the foundation of the two second-generation instruction releases, and a starting point for anyone fine-tuning a small Kiswahili model.
- Parameters
- 149.0M
- 173.6M stored; embeddings untied
- Layers
- 20
- 768 wide, feed-forward 2,048
- Context
- 2,048
- tokens
- Licence
- Apache 2.0
How it was trained.
- Data
- 2.62B tokens, 3.74M documents, eight registers: web, news, encyclopedia, translated educational, parliamentary, religious, long-form, chat format
- Tokens seen
- 1.97B, 75.2% of one pass
- Architecture
- Pre-norm decoder: RMSNorm with QK-norm, SwiGLU, RoPE, grouped-query attention (12 query, 3 key-value heads), Canon layers, document-boundary masking
- Schedule
- Warmup-stable-decay, decay completed over the final 10%
- Optimiser
- Muon for matrices, AdamW for embeddings, norms and head
- Hardware
- Kaggle TPU v5e-8, about 27 TPU-hours
- Validation loss
- 3.1560 on the held-out split
How it measured.
News is in the pretraining mix and the test articles were not checked for overlap, so the bits-per-byte row is in-domain. Belebele and AfriXNLI do not separate any model in the comparison from chance.
- Bits per byte, 120 MasakhaNEWS test articles
- 0.926 (XGLM-564M 1.122, BLOOM-560m 1.699, Qwen2.5-0.5B 2.393)
- MasakhaNEWS-sw topic accuracy, n = 476
- 73.1% [69.0, 76.9] (XGLM-564M 72.5%)
- Belebele-sw, n = 900
- 29.3%, chance 25%
- AfriXNLI-sw, n = 600
- 33.7%, chance 33.3%
What it does, and what it gets wrong.
Does well
- Fluent Kiswahili continuation
- The lowest bits per byte in its comparison, on in-domain news
- News topic classification at the level of models three to four times its size
- Weight-only int8 brings it from 696 MB to 251 MB, moving bits per byte by 0.0001 at most
Gets wrong
- Not an instruction model: it ignores constraints and stops only about half the time
- States confident falsehoods; very little room for world knowledge
- Primarily Tanzanian Kiswahili; part of the corpus is machine-translated
- No safety tuning
Its place in the family.
- KW5-109MFirst generation · BaseSucceeds the first-generation base, whose learning-rate decay never ran.
- KW5-149M-instructSecond generation · InstructInstruction-tuned into KW5-149M-instruct.
- KW5-149M-instruct-r2Second generation · Instruct r2Instruction-tuned again into KW5-149M-instruct-r2.
- Second generation · baseKW5-149MTwenty layers with Canon layers and grouped-query attention, pretrained on 1.97B tokens of Kiswahili. It has the lowest bits per byte in its comparison: 0.926 on MasakhaNEWS test text, a domain it was trained on.
- Second generation · instruct r2KW5-149M-instruct-r2The revised instruction release. Given a passage, it answers 43.2% of questions over real text and declines 55% of the questions it cannot answer.
- Second generation · instructKW5-149M-instructThe first fine-tune of the second generation: two epochs over 465M tokens. Kept for comparison, with its repetition documented.
- First generation · baseKW5-109MA Llama-compatible decoder, twelve layers, trained on one T4 in about 60 GPU-hours. Its learning rate never decayed; the second generation was built to fix that.
- First generation · instructKW5-109M-instructA merged LoRA fine-tune, and the baseline every later instruction model is measured against.