KW5-109M-instruct. The baseline.
The first instruction release: a LoRA fine-tune of KW5-109M, and the baseline every later instruction model is measured against.
It writes fluent Kiswahili in answer form, but answers from memory rather than from a supplied passage and almost never declines. Its numbers are the floor the second generation is measured from.
- Parameters
- 109.5M
- LoRA merged into the base
- Reading a passage
- 2.4%
- n = 250
- Declining the unanswerable
- 2.5%
- n = 40
- Licence
- Apache 2.0
How it was trained.
- Base
- KW5-109M, step 6,150
- Method
- LoRA rank 16, alpha 32, on attention and feed-forward projections; merged
- Data
- About 107,000 Kiswahili instruction and conversation examples, 137 of them teaching it to say it does not know
- Hardware
- One NVIDIA T4
How it measured.
The reading prompts are short (median 146 tokens), so the low score is not a context-length effect: the model ignores the passage.
- Answer a question from a real passage, n = 250
- 2.4%
- Decline a question nobody could answer, n = 40
- 2.5%
- Answer an easy question and get it right, n = 30
- 20.0%
- List exactly N items, n = 16
- 43.8%
What it does, and what it gets wrong.
Does well
- Fluent Kiswahili in answer form
- A fixed baseline for the instruction suite
Gets wrong
- Does not use a supplied passage
- Almost never declines, so it invents answers
- Facts are unreliable
- Minimal safety tuning; not for unsupervised use
Its place in the family.
- KW5-109MFirst generation · BaseBuilt on the first-generation base.
- KW5-149M-instructSecond generation · InstructIts 1-in-40 declining is what the second generation set out to fix.
- KW5-149M-instruct-r2Second generation · Instruct r2KW5-149M-instruct-r2 reads passages 18 times as often.
- Second generation · baseKW5-149MTwenty layers with Canon layers and grouped-query attention, pretrained on 1.97B tokens of Kiswahili. It has the lowest bits per byte in its comparison: 0.926 on MasakhaNEWS test text, a domain it was trained on.
- Second generation · instruct r2KW5-149M-instruct-r2The revised instruction release. Given a passage, it answers 43.2% of questions over real text and declines 55% of the questions it cannot answer.
- Second generation · instructKW5-149M-instructThe first fine-tune of the second generation: two epochs over 465M tokens. Kept for comparison, with its repetition documented.
- First generation · baseKW5-109MA Llama-compatible decoder, twelve layers, trained on one T4 in about 60 GPU-hours. Its learning rate never decayed; the second generation was built to fix that.
- First generation · instructKW5-109M-instructA merged LoRA fine-tune, and the baseline every later instruction model is measured against.