KW5-109M. Where the family began.
The first KW5 model: 109.5M parameters, pretrained from scratch on one T4.
A Llama-compatible base model that writes fluent Kiswahili and gets facts wrong. Its learning-rate decay never ran, and the second generation was planned around fixing that. It stays public as the baseline the family grew from.
- Parameters
- 109.5M
- tied embeddings
- Layers
- 12
- 768 wide, feed-forward 2,048
- Context
- 1,024
- tokens
- Licence
- Apache 2.0
How it was trained.
- Data
- FineWeb-2 Kiswahili, cleaned and deduplicated, about 705M unique tokens
- Tokens seen
- 1.41B, two passes
- Tokenizer
- 32,000-piece SentencePiece BPE, trained on Kiswahili
- Schedule
- Warmup-stable-decay, stopped in the stable phase: the decay was set for a token budget about four times too large
- Optimiser
- 8-bit AdamW, learning rate 3e-4
- Hardware
- One NVIDIA T4, about 60 GPU-hours
How it measured.
Both are consistent with chance, and with 50 items they are a floor, not a measure of skill. Its one real skill is fluent continuation.
- Belebele-sw, n = 50
- 32%, chance 25%
- AfriXNLI-sw, n = 50
- 32%, chance 33%
What it does, and what it gets wrong.
Does well
- Fluent Kiswahili continuation
- Runs anywhere a Llama-architecture model runs
Gets wrong
- Continues text; does not answer questions
- Facts are unreliable
- Trained across document boundaries with no attention mask
- No safety tuning
Its place in the family.
- KW5-109M-instructFirst generation · InstructFine-tuned into KW5-109M-instruct.
- KW5-149MSecond generation · BaseSucceeded by KW5-149M, which completed the decay this run never reached.
- Second generation · baseKW5-149MTwenty layers with Canon layers and grouped-query attention, pretrained on 1.97B tokens of Kiswahili. It has the lowest bits per byte in its comparison: 0.926 on MasakhaNEWS test text, a domain it was trained on.
- Second generation · instruct r2KW5-149M-instruct-r2The revised instruction release. Given a passage, it answers 43.2% of questions over real text and declines 55% of the questions it cannot answer.
- Second generation · instructKW5-149M-instructThe first fine-tune of the second generation: two epochs over 465M tokens. Kept for comparison, with its repetition documented.
- First generation · baseKW5-109MA Llama-compatible decoder, twelve layers, trained on one T4 in about 60 GPU-hours. Its learning rate never decayed; the second generation was built to fix that.
- First generation · instructKW5-109M-instructA merged LoRA fine-tune, and the baseline every later instruction model is measured against.