KW5-230M: the third generation, Kiswahili first

Today we roll out KW5-230M, the third generation of our Kiswahili language models: a base model and an assistant, trained from the tokenizer up on Kiswahili text and open under Apache 2.0. Both are public on Hugging Face, kw5-230M and kw5-230M-instruct, with the modelling code and the line encoder beside the weights.

Ask the second generation what 7 plus 5 is and it would tell you it was multiplying. Ask the third:

Hesabu: 71 - 17 + 20 = ?

<mawazo>Tunafuata mpangilio wa hesabu: mabano kwanza, kisha kugawanya na
kuzidisha kutoka kushoto kwenda kulia, kisha kujumlisha na kutoa kutoka
kushoto kwenda kulia.
Tunatoa 71 - 17. Tunatoa tarakimu kuanzia kulia: 1 ni ndogo kuliko 7,
tunakopa kumi: 11 - 7 = 4; 7 - 1 tuliyokopa - 1 = 5. Kwa hiyo 71 - 17 = 54.
Sasa tuna: 54 + 20.
Tunajumlisha 54 + 20. Tunajumlisha tarakimu kuanzia kulia: 4 + 0 = 4,
tunaandika 4; 5 + 2 = 7, tunaandika 7. Kwa hiyo 54 + 20 = 74.</mawazo>
Jibu ni 74.

That is the reply exactly as the model wrote it in the evaluation run. It was pretrained on 7.90B tokens of Kiswahili on a Kaggle TPU, and at 230M parameters it is small enough to run on hardware an institution already owns.

It works the sum. 91.4% on bare arithmetic up to four digits, up from 2.3%, and 70.5% on expressions with brackets. The working sits in a <mawazo> block, and only arithmetic gets one, so a plain question is answered without reasoning in front of it.

It remembers the conversation. Told a fact three to nine turns earlier, it recalls it 98.8% of the time. Asked about something it was never told, it says so: Hujaniambia mji wowote.

It reads before it answers. 58.4% on questions over real Swahili passages, and 70.0% correct declines when the passage does not hold the answer.

v3 started by distrusting the second generation's numbers. An audit found its reading test always asked for the second answer, its sums were 0 of 60, and its refusals named the answer they refused; the blog post has the details. So the third generation was scored on items no training data comes near, against bars fixed before training, with gates that stopped the run on a fault. They caught several, a fine-tune at the wrong rotary frequency among them, and each is in the run log with its fix.

We set fourteen bars on 15 October, the day before training began. Seven are met. The assistant still keeps the first version of a list after it has changed, still writes the wrong sum for many two-step word problems, and declined four easy questions it should have answered. Every miss is in the model card with its number and its interval.

Language is infrastructure. A ministry, a hospital or a school that wants an assistant in Kiswahili should not have to send every question abroad to get one, or accept a model that treats the language as an afterthought. KW5 is the research behind that: a model trained on the language it is used in, measured in the open, small enough to sit behind Cordon on your own machines.

The full account, from how the size was chosen to what the gates caught, is in the technical report.