KW5-149M-instruct-r2. The release to try first.
The revised instruction release: it answers from a supplied passage and declines about half of the questions it cannot answer.
The KW5 release to try first. It is for research and for Kiswahili tasks where the answer is in the prompt and someone checks the output: questions about a passage, rewriting, short explanations.
- Parameters
- 149.0M
- 173.6M stored
- Reading a passage
- 43.2%
- n = 250, [37.2, 49.4]
- Declining the unanswerable
- 55.0%
- n = 40, [39.8, 69.3]
- Licence
- Apache 2.0
How it was trained.
- Base
- KW5-149M, 1.97B tokens, decay completed
- Fine-tune
- Full-parameter, 2 epochs, 1,944 steps, 254.8M tokens
- Optimiser
- AdamW, learning rate 1e-5, 3% warm-up
- Context
- 1,024 tokens, loss on assistant turns only
- Hardware
- Kaggle TPU v5e-8
How it measured.
The same frozen 924-item Kiswahili suite is run on every KW5 instruction model, greedy, each in its own chat format.
- Answer a question from a real passage, n = 250
- 43.2% (KW5-149M-instruct 39.2%, KW5-109M-instruct 2.4%)
- Pick the right one of two facts in a passage, n = 48
- 54.2% (27.1%, 2.1%)
- Say the passage does not contain the answer, n = 60
- 33.3% (5.0%, 15.0%)
- Decline a question nobody could answer, n = 40
- 55.0% (42.5%, 2.5%)
- Follow-up question in a conversation, n = 8
- 37.5% (12.5%, 12.5%)
- Arithmetic, n = 128
- 2.3%
What it does, and what it gets wrong.
Does well
- Reading a supplied passage and answering from it
- Declining about half of unanswerable questions
- Stopping on its own for short answers
Gets wrong
- Arithmetic is near zero
- Facts are unreliable; not for factual lookup
- Structured answers sometimes carry English
- No real safety tuning: do not put it in front of end users without your own safeguards
Its place in the family.
- KW5-149MSecond generation · BaseBuilt on the second-generation base.
- KW5-149M-instructSecond generation · InstructRevises the first second-generation fine-tune.
- KW5-109M-instructFirst generation · InstructMeasured against the first instruction release.
- Second generation · baseKW5-149MTwenty layers with Canon layers and grouped-query attention, pretrained on 1.97B tokens of Kiswahili. It has the lowest bits per byte in its comparison: 0.926 on MasakhaNEWS test text, a domain it was trained on.
- Second generation · instruct r2KW5-149M-instruct-r2The revised instruction release. Given a passage, it answers 43.2% of questions over real text and declines 55% of the questions it cannot answer.
- Second generation · instructKW5-149M-instructThe first fine-tune of the second generation: two epochs over 465M tokens. Kept for comparison, with its repetition documented.
- First generation · baseKW5-109MA Llama-compatible decoder, twelve layers, trained on one T4 in about 60 GPU-hours. Its learning rate never decayed; the second generation was built to fix that.
- First generation · instructKW5-109M-instructA merged LoRA fine-tune, and the baseline every later instruction model is measured against.