KW5-230M: a Kiswahili assistant from the tokenizer up

The second KW5 generation could continue Kiswahili fluently and could not do much else. Its revised instruction model got 3 of 128 sums right, declined about half of the questions nobody could answer, and refused a follow-up while naming the right answer inside the refusal. The brief for the third was specific: hold facts and context through a conversation, do whole-number arithmetic and order of operations, answer from a passage, and say so when the passage or the conversation does not hold the answer. All of it on 60 hours of TPU time, at a size that clears the Chinchilla ratio [1].

This report is what came out, measured against bars set before any of it was trained.

v3 began with an audit of the second generation, run against a harness and items we had not written. The published grounded test always asked for the second of two candidates, so it measured a positional habit; asked for the first, the same models scored 70.8% where the card said 50.0% and 37.5%. Several test items shared the training generator's templates, one of them word for word. Asked a capital directly, the best model refused none of eight; asked the same capitals as a follow-up, it refused six, naming the right answer inside the refusal. Arithmetic was 0 of 60, on plain phrasings and on the generator's own. On 924 new items, every published model read a real passage less than half the time and scored at floor on sums.

The audit also found that the tokenizer split every word that follows a newline into its word-internal pieces, so the first word of every prompt had been fragmented in training and in use. What it found set the brief for v3: an evaluation the training data cannot reach, bars fixed in advance, text encoded line by line, and gates that stop a run. The engineering account is in The audit that started KW5 v3.

On 8 October we froze a second evaluation suite of 540 items, beside the 924 already in use, and on 15 October, the day before a token of v3 was trained, fourteen release bars. Templates, names, places and wording in the new suite appear in no data generator, a test checks that, and both suites are hash-checked every time they run. A missed bar is reported as missed. It is not moved.

The rule: the largest shape that reaches 30 training tokens per stored parameter, embeddings counted, in the pretraining hours. A throughput sprint on the TPU measured three deep, thin shapes in the MobileLLM pattern [2]. The 230M shape (24 layers, 896 wide, 14 query heads on 2 key-value heads, tied embeddings) ran at 66,013 tokens a second in the sprint and 57,109 in the first session's calibration, and that cleared the rule with 34.3 tokens per parameter. The budget, 7,895,777,280 tokens, was locked in that session and stored in every checkpoint, so no later session could quietly change it.

The block is the second generation's, unchanged: pre-norm RMSNorm with QK-norm, SwiGLU, rotary positions, document-boundary masking, and Canon layers, short causal convolutions that give each block a local memory beside attention. Changing the block and the data in the same run would have left us unable to say which one mattered.

5.58B tokens: the audited second-generation corpus across eight registers, about 2.5B tokens of translated educational web text, long-form PDFs, Swahili FineWiki, and two registers made by script, arithmetic with its working and long conversations. Every equation in the generated data is checked by a test. The last 15% of training ran at context 2,048 on a mix weighted toward the encyclopedia (three times), arithmetic (twice) and chat (four times), so that fine-tuning would start close to its target.

One fix ran through all of it. SentencePiece gives a word its word-initial piece only after a space, so every word that follows a newline had been fragmented in the second generation's corpus and in its chat format, the first word of every prompt included. v3 encodes line by line, and the encoder ships beside the weights so that anyone running the model encodes exactly as it was trained.

Pretraining stopped itself on failure. A checkpoint past step 200 had to reload in a fresh process and reproduce its loss before training continued: the second generation lost about 27 TPU-hours to a reload path nobody had tested. Greedy samples were printed every 500 steps and read as text, not as numbers.

The gates caught real faults, and the faults are on the record. The first fine-tune trained at the wrong rotary frequency because the loader took the base's weights and not its recorded positions; it was retrained, and the wrong checkpoints deleted. A teacher job that had finished was rerun for days because progress was counted in lines, not distinct conversations. A decay session died at its first validation because twenty batches were traced into one program. Each is a line in the run log with its date and its fix.

Script generation cannot write natural conversation, so a teacher model translated some. Its Kiswahili was measured before any of it was used: on 200 segments of WMT24++ English to Swahili, against the professional post-edit, Gemma 4 26B-A4B scored chrF++ 58.3, the 12B 57.4 and E4B 54.0 (two human references agree at 89.4). Read by hand, the 26B writes fluent ordinary prose and fails on idioms and slang: "good riddance" came back as "it could not be heard". Throughput, not quality, was the limit. At about 98 tokens a second on two T4s, twenty GPU-hours buy roughly ten thousand conversations, so the teacher was spent only where scripts cannot reach.

Fine-tuning ran twice. The first pass trained on a mix read source by source: instruction sets with their worst rows cut, grounded questions over real passages, generated arithmetic and conversations, and about 10,700 conversations translated by Gemma 4 26B-A4B, kept only when every turn passed a check. Then the first fine-tune was sampled eight times per prompt. Questions it got right at most once in eight became declines; questions it got right six or more times became answers, as R-Tuning does by likelihood [3]. Replies that passed a verifier (right number, every equation true, span present, exact count) joined the second mix. The assistant is that second pass: 204,483 conversations, two epochs, 3,195 steps from the base.

The first pass met six of the fourteen bars, and its misses were read reply by reply before the second mix was built. They were coverage, not capacity: sums phrased as imperatives the arithmetic generator never used ("Toa N kutoka N" scored 0 of 7), two-step word problems answered with one operation, a list the assistant never learned to track because the training dialogues always repeated each change back, and swapped-passage negatives in a format the mix only ever answered. Each miss got a generator, tested, with the suite's own wording kept out by test, and the calibration declines were capped and balanced with 1,200 answered plain facts so that declining would not spread to what the model knows. The second pass trained for 4.4 hours on the TPU, without a gate stop, and exported with a reload difference of zero.

DPO was planned behind a gate: it would run only if the second pass missed a bar on format, persistence or refusal, and its preference pairs would come from the first pass's samples, off-policy for the model it tuned. This release ships the second pass without it. GRPO was ruled out at the plan stage, since it needs fast sampling inside the training loop that neither the TPU trainer nor a T4 provides. The Kiswahili safety sets, XSTest and MultiJail, have not been run; they are the first measurements after release.

Axis	KW5-230M-instruct230M	swahili-gemma-1b1B	Qwen3-0.6B596M	Qwen2.5-0.5B-Instruct494M
Bare arithmeticn = 128	91.4met	70.3	41.4	31.3
Order of operationsn = 200	70.5met	44.5	83.0	57.0
Word problemsn = 24	62.5	66.7	16.7	0.0
Word problems, unseenn = 120	24.2missed	39.2	0.8	2.5
AfriMGSMn = 100	1.0missed	8.0	3.0	2.0
Read a real passagen = 250	58.4missed	58.4	24.4	16.8
Pick the right factn = 48	77.1	70.8	16.7	52.1
Say the passage lacks itn = 60	70.0met	1.7	0.0	0.0
Recall across turnsn = 80	98.8met	16.3	0.0	10.0
Say it was never toldn = 40	82.5met	2.5	0.0	0.0
A list that changedn = 40	5.0missed	2.5	2.5	2.5
A balance across turnsn = 40	2.5missed	0.0	0.0	0.0
An instruction that persistsn = 20	35.0missed	25.0	0.0	0.0
Follow-upn = 8	87.5met	75.0	0.0	0.0
Decline the unanswerablen = 40	97.5met	2.5	0.0	0.0
Answer the answerablen = 30	86.7missed	100.0	100.0	100.0
TruthfulQA MC1n = 200	26.5	26.5	20.0	20.5
List exactly Nn = 16	56.3	50.0	18.8	12.5

The frozen Kiswahili suites, 1,464 items, greedy, each model in its own chat template. A green tick in the first column is a release bar, frozen on 15 October before training; KW5-230M-instruct meets 7 of the 14. The conversation and declining axes are scored against the replies a Kiswahili assistant should give, which KW5 was trained to give; read those columns with that in mind.

Beside other small models on the same 1,464 items, it scores 54.4% overall, against 38.0% for swahili-gemma-1b, 25.4% for Qwen3-0.6B and 21.1% for Qwen2.5-0.5B-Instruct. That gap is partly a measurement of voice. The conversation and declining items are scored against the replies a Kiswahili assistant should give, and KW5 was trained to give them in that voice. On the 1,116 items whose answer is a number, a span or a choice, it leads the 1B model 51.4% to 46.7%, and loses word problems and AfriMGSM to it.

On public benchmarks, the assistant scores 35.6% on Belebele [4], where every model beats chance and its interval overlaps the next model's. No model in the comparison clears chance on AfriXNLI. MasakhaNEWS, at 73.1%, is in the training domain.

It keeps the first version of a list after the list has changed (2 of 40). It works any sum written down, and often writes the wrong sum for a two-step problem. It declined four easy questions it should have answered, among them when Tanganyika became independent: the calibrated declines reached some plain facts. These are coverage failures, not capacity ones, and they say what the next mix needs.

One caveat travels with every number above. The second mix was built after the first fine-tune was scored on the same suite, to cover what it missed. The items themselves were kept out by hash and by test, but the suite's arithmetic is no longer novel in frame, only in wording. The next measurement is a suite written after this release, and the safety sets (XSTest and MultiJail in Kiswahili), which have not been run.

KW5-230M and KW5-230M Instruct, under Apache 2.0, with the modelling code and the line encoder beside them. The benchmark run ships with this page: results, significance tests and the manifest that pins every model's revision.