KW5-149M-instruct. The first that learned to decline.
The first instruction release of the second generation, the one that learned to decline.
It answers in Kiswahili, stops when it is finished and declines some questions it cannot answer. KW5-149M-instruct-r2 replaces it; it stays public so the improvement can be checked.
- Parameters
- 149.0M
- 173.6M stored
- Reading a passage
- 39.2%
- n = 250, [33.4, 45.4]
- Declining the unanswerable
- 42.5%
- n = 40
- Licence
- Apache 2.0
How it was trained.
- Base
- KW5-149M, 1.97B tokens, decay completed
- Fine-tune
- Full-parameter, 2 epochs, 3,550 steps, 465.3M tokens
- Mix
- About 57% instruction following, 20% reasoning, 10% abstention, with grounded QA, summarisation and hand-curated Kiswahili sets
- Optimiser
- AdamW, learning rate 1e-5, 3% warm-up
- Forgetting
- Held-out pretraining loss 3.2031 before, 3.2640 after: the base ability is intact
- Hardware
- Kaggle TPU v5e-8
How it measured.
Its declining is learned phrasing, not reasoning about uncertainty: the abstention data was templated, and the refusals are close to word for word.
- Answer a question from a real passage, n = 250
- 39.2%
- Decline a question nobody could answer, n = 40
- 42.5%
- Answer an easy question and get it right, n = 30
- 56.7%
- List exactly N items, n = 16
- 12.5%
- Arithmetic, n = 128
- 0.8%
What it does, and what it gets wrong.
Does well
- Declines some unanswerable questions, and says why
- Stops on its own: 0.94 of replies at a 160-token cap
- Follows simple constraints such as one sentence
Gets wrong
- Repetition on longer generations is documented in the card
- Declining is not reliable and is not a confidence estimate
- Facts are unreliable
- No real safety tuning
Its place in the family.
- KW5-149MSecond generation · BaseBuilt on the second-generation base.
- KW5-149M-instruct-r2Second generation · Instruct r2Revised as KW5-149M-instruct-r2.
- KW5-109M-instructFirst generation · InstructDeclines 17 of 40 where the first instruction release declined 1.
- Second generation · baseKW5-149MTwenty layers with Canon layers and grouped-query attention, pretrained on 1.97B tokens of Kiswahili. It has the lowest bits per byte in its comparison: 0.926 on MasakhaNEWS test text, a domain it was trained on.
- Second generation · instruct r2KW5-149M-instruct-r2The revised instruction release. Given a passage, it answers 43.2% of questions over real text and declines 55% of the questions it cannot answer.
- Second generation · instructKW5-149M-instructThe first fine-tune of the second generation: two epochs over 465M tokens. Kept for comparison, with its repetition documented.
- First generation · baseKW5-109MA Llama-compatible decoder, twelve layers, trained on one T4 in about 60 GPU-hours. Its learning rate never decayed; the second generation was built to fix that.
- First generation · instructKW5-109M-instructA merged LoRA fine-tune, and the baseline every later instruction model is measured against.