nkthebass/tinybrainbot-350mV3-math

TinyBrainBot 350M V3 — Math

A 348M-parameter decoder-only model fine-tuned to do column arithmetic with a written-out scratchpad. Successor to the TinyBrainBot 320M V2 Math.

TL;DR: 99.4% average across the nine GPT-3 arithmetic sub-tasks, beating the 320M V2 Math (91.5%). Adds cleanly to 14 digits, does 3-digit multipliers (98.0%), and handles negative results (85% overall, but only 58% at 5 digits - see below). It is a calculator with a scratchpad, not a word-problem solver — see the honest limits below.

Updated 2026-09-08. This revision fixes three defects present in the first release: negative results returned confident wrong positives (2.0% → 85%), addition silently dropped a column past 8 digits (widest clean sum 8 → 14), and 3-digit multipliers had zero training examples (8.3% → 98.0%). The cost is ~0.5 points of ordered-subtraction accuracy and 4 points of one-step word problems; both are itemised below.

---

Model details

Parameters	348,342,912 (~348M)
Architecture	Decoder-only, pre-norm, RMSNorm, SwiGLU, RoPE, GQA (18 Q / 6 KV heads)
Hidden size / layers	1152 / 22
Context length	2048
Vocabulary	32,000 (tbb-32k-v2, tied embeddings)
Lineage	350M V3 Base → Instruct → 10 rounds of math SFT → 3 merged LoRA adapters

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

m = "nkthebass/tinybrainbot-350mV3-math"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m)

msgs = [{"role": "user", "content": "What is 4827 plus 3956?"}]
p = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(p, return_tensors="pt"), max_new_tokens=200, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Use apply_chat_template rather than hand-building the prompt — this family's template is space-separated, not newline-separated, and getting it wrong degrades output badly.

Greedy decoding (do_sample=False) is strongly recommended. The scratchpad is a deterministic routine; sampling introduces digit errors mid-column.

Evaluation — arithmetic

GPT-3 arithmetic protocol, zero-shot, n=300 per sub-task, greedy, exact match on the final number. Identical problems and seed for both models.

Sub-task	350M V3 Math	320M V2 Math
2-digit addition	100.0	83.0
3-digit addition	100.0	97.7
4-digit addition	100.0	98.7
5-digit addition	100.0	91.7
2-digit subtraction	99.3	95.3
3-digit subtraction	98.3	91.0
4-digit subtraction	98.3	94.7
5-digit subtraction	99.0	92.0
2-digit multiplication	100.0	79.3
Average	99.43	91.5

The four subtraction cells sit at 98.3–99.3 rather than 100. That is the measured price of teaching the routine to produce negative results: the model must now decide which operand is larger, and that decision sometimes misfires on ordered operands too - producing a negative answer to a question whose answer is positive.

The errors are all in the comparison, not the arithmetic. Independent testing binned ordered pairs by how close b is to a and found the wrong-sign rate rising smoothly as they converge (12.5% at b/a 0.98–1.00, falling to 0.0% below 0.30), with correct-answer rate equal to exactly 100% − wrong-sign in every bin. Zero arithmetic errors underneath.

That points at a single learned magnitude judgement with no digit-scan procedure behind it - the same shape of problem the column routine already solved once for addition and subtraction. It is a fixable gap, not a floor. Teaching comparison as an explicit left-to-right digit scan, with close-operand and 5-digit pairs in the generator, is the obvious next round.

→ 9/9 sub-tasks improved.

Important: these operands are ordered, so no answer is negative

The harness orders subtraction operands (if b > a: a, b = b, a), so every answer in the table above is non-negative. The GPT-3 protocol as originally defined samples operands independently and permits negative results, so this table is not directly comparable to published GPT-3 numbers. The figure above is 99.43% on non-negative subtraction.

Negative results are measured separately, and in this revision they work:

Operand width	first release	this revision
1-digit	—	100.0%
2-digit	—	88.3%
3-digit	—	98.3%
4-digit	—	80.0%
5-digit	—	58.3%
mean, widths 1–5 (n=60 each)	~2%	85.0%

Accuracy falls off with operand width, and the failure is the magnitude comparison, not the column routine - the model decides which operand is larger with what appears to be a single learned judgement rather than a digit scan, and that judgement degrades as the numbers get longer and closer together. Independent testing found the same shape, with sign-detection dropping sharply at 5 digits and errors concentrating where b/a approaches 1.

An earlier version of this card reported 92.7% here. That figure came from a harness that sampled operand width from choice([1,2,3,4]) - it never tested 5 digits, which is exactly where the weakness lives. The table above samples all five widths.

USER: What is 32 minus 83?
BOT : Subtracting. 83 is larger than 32, so the answer is negative. Compute 83 - 32.
      ones: 3 - 2 = 1. tens: 8 - 3 = 5. That gives 51. The answer is -51.

The first release returned 29 for that question — a confident, well-formatted, wrong positive number, because every training generator ordered its operands and so did the benchmark. The fix states the sign decision out loud and then reuses the existing routine on the swapped pair.

Does it generalize, or did it memorize?

The 99.43% above is in-distribution by construction. These probes are the honest test, and the phrasings are held out of training by construction so they cannot be re-fitting.

Probe	first release	this revision	Reading
Held-out phrasings (n=60) "Sum 63 and 29.", "Deduct 18 from 62."	80.0%	81.7%	Question parsing was a bigger bottleneck than arithmetic. An earlier build measured 33.3% here before a phrasing-focused adapter.
Widest clean addition	8 digits	14 digits	See below - the limit was vocabulary, not arithmetic.
7-digit addition	100%	100%	Comfortably inside the new range.
3×3 multiplication (n=200)	8.3%	98.0%	The training set contained zero examples with a 3-digit multiplier, so the old 8.3% was extrapolation to an unseen width. It now folds partial products pairwise through the column routine instead of asserting the three-way sum.

The addition ceiling was a vocabulary problem

The training scratchpad only ever names six place values (ones … hundred-thousands). The first release learned the sequence and invented two more on its own - millions and ten-millions appear in no training example - so it added 7- and 8-digit numbers correctly and then fell off a cliff:

Width	6	7	8	9	10	12	14	16	18
first release	100	100	100	0	0	0	0	0	0
this revision	100	100	100	100	95	100	90	65	25

At 9 digits the first release had no name for the ninth column, skipped it, and returned an answer exactly one digit short - with every column it did compute arithmetically perfect. Across widths 9-18 essentially every failure was of that kind, not an arithmetic error.

Extending the place-name list from 6 entries to 19 moved the clean ceiling from 8 to 14, and the skipped-column failures drop to zero through 14. The residual failures at 16-18 are genuine arithmetic slips in a long carry chain, which is a different and much more ordinary problem. Keep addition to 14 digits or fewer.

Evaluation — word problems

Straight arithmetic is what this model was built for; word problems are the harder, independent test. None of these datasets appear in any TinyBrainBot training set.

Five independent, human-written sets. Zero-shot, greedy, exact match on the final number.

Benchmark	350M V3 Math	n	Structure
ASDiv	16.5	200	mostly single-step
MultiArith	7.8	180	2–3 operations
SVAMP	5.0	200	one-step, adversarially perturbed
MAWPS	5.0	200	one/two-step
GSM8K	4.0	200	multi-step, free-form
One-step, trivial (n=100)	79.0	100	two numbers, one operation

The spread tracks problem structure, and that is the most honest thing on this card: the model handles a trivial one-step problem most of the time (79.0), converts one sentence into one operation reasonably often (ASDiv 16.5), and essentially cannot chain operations (GSM8K/MAWPS 4.0-5.0). Single digits on GSM8K is the honest expectation at 348M — the 320M V2 Math card reports ~3–4%.

One-step accuracy fell from 83.0 in the first release. The measured cause is operation selection, not arithmetic: "drops in 836 more" gets read as subtraction. Adapters that train the column routine harder make the model reach for it more often, including when the wrong operands are chosen.

The 99.4% and the 4% are not a contradiction

They measure two different mechanisms. The column routine fires on the arithmetic question form; inside a word problem the model falls back to mental arithmetic, which is not reliable. An actual generation from the GSM8K run:

<think> The total ratio representing their ages was 7+11=181 ... </think>

7 + 11 = 181 — concatenation instead of addition, from a model that does 5-digit column subtraction at 99.7%. Treat the arithmetic score as a measure of a routine, not of numeracy. If you need a correct sum, ask for it directly.

Sample output

Verbatim from the F16 GGUF in this repo, greedy (--temp 0), unedited:

USER: What is 4827 plus 3956?
BOT : Adding. ones: 7 + 6 = 13, write 3 carry 1. tens: 2 + 5 + 1 (carry) = 8, write 8.
      hundreds: 8 + 9 = 17, write 7 carry 1. thousands: 4 + 3 + 1 (carry) = 8, write 8.
      The answer is 8783.

USER: What is 90210 minus 4877?
BOT : Subtracting. Pad 4877 to 04877. ones: 10 - 7 = 3, borrow 1. tens: 10 - 7 = 3, borrow 1.
      hundreds: 11 - 8 = 3, borrow 1. thousands: 9 - 4 = 5. ten-thousands: 8 - 0 = 8.
      The answer is 85333.

USER: What is 47 times 38?
BOT : Multiplying. Break 38 into digits. 47 x 8 = 376. 47 x 3 x 10^1 = 1410.
      Add 376 + 1410. Adding. Pad 376 to 0376. ones: 6 + 0 = 6, write 6.
      tens: 7 + 1 = 8, write 8. hundreds: 3 + 4 = 7, write 7. thousands: 0 + 1 = 1, write 1.
      The answer is 1786.

Note the third one: rather than asserting the sum of the partial products, it runs the column addition routine on them. That nesting is deliberate - asserting the sum was a measured failure mode in earlier rounds.

Reasoning tags on word problems

For word problems (not straight arithmetic) the model emits a <think> ... </think> block before its answer. Measured over 40 GSM8K generations: 100% of blocks were opened and closed correctly, with the answer outside the block.

<think> 3 boxes with 24 pencils each: multiply 3 * 24. 3 * 24 = 72.</think> The answer is 72.

These are ordinary text, not special tokens. <think> is not in added_tokens.json, so skip_special_tokens=True will not remove it and it will appear in your output.

llama.cpp's reasoning extraction does not work with this model — tested on b10034, both --reasoning-format deepseek and -rea on leave reasoning_content empty and the tags in content. llama.cpp only engages that parser for chat-template formats it recognises, and this model's template falls back to generic. Strip the block yourself:

import re
answer = re.sub(r"<think>.*?</think>", "", raw, flags=re.S).strip()

Because closure is reliable, that one line is sufficient. Do not add <think>/</think> to the tokenizer as special tokens to try to fix this — the model emits them as a multi-token character sequence and has no probability mass on any new single token, so registering them would break generation rather than improve it.

Worked examples

All verbatim from the F16 GGUF in this repo, greedy (--temp 0), unedited. These are selected: 7 of 12 attempts were correct. See the failure mode below - the successes and failures split on a single, visible criterion.

Borrowing across a run of zeros:

USER: There were 15000 votes and 6842 were rejected. How many counted?
BOT : <think> Start with 15000. Then subtract 6842. Subtract 15000 - 6842 column by column:
      ones: 10 - 2 = 8, borrow 1. tens: 9 (after borrow) - 4 = 5, borrow 1.
      hundreds: 9 (after borrow) - 8 = 1, borrow 1. thousands: 14 (after borrow) - 6 = 8,
      borrow 1. ten-thousands: 0 (after borrow) - 0 = 0. So 15000 - 6842 = 8158.</think>
      The answer is 8158.

Multiplication, with the column-addition routine run on its own partial products:

USER: Each crate holds 24 bottles. How many bottles are in 15 crates?
BOT : <think> 24 bottles per crate: multiply 15 * 24. Multiply 15 * 24 using partial
      products. Split 24 into 20 and 4. 15 * 4 = 60. 15 * 20 = 300. Add 60 + 300:
      ones: 0 + 0 = 0. tens: 6 + 0 = 6. hundreds: 0 + 3 = 3. So 15 * 24 = 360.</think>
      The answer is 360.

Recognising "how many more" as subtraction:

USER: A shop sold 3672 items in May and 2145 in June. How many more in May?
BOT : <think> ... ones: 12 - 5 = 7, borrow 1. tens: 6 (after borrow) - 4 = 2.
      hundreds: 6 - 1 = 5. thousands: 3 - 2 = 1. So 3672 - 2145 = 1527.</think>
      The answer is 1527.

Addition with carry, and a non-math question showing the format generalises:

USER: A library had 8340 books and received 1275 more. How many books now?
BOT : <think> ... tens: 4 + 7 = 11, write 1 carry 1. hundreds: 3 + 2 + 1 (carry) = 6.
      thousands: 8 + 1 = 9. So 8340 + 1275 = 9615.</think> The answer is 9615.

USER: What is the capital of Japan?
BOT : <think> The capital of Japan is Tokyo.</think> The answer is Tokyo.

The failure mode is visible in the first few words

USER: A field is 340 metres long and 27 metres wide. What is its area?
BOT : <think> First, calculate the area of the field by multiplying its length by its width:
      340 m * 27 m = 1080 m2.</think> The answer is 1080.        WRONG (9180)

Every correct answer above says "multiply X * Y" or "subtract X - Y column by column" and then runs the routine. Every wrong one opens with "First, calculate..." and asserts a number in prose. Same model, same class of question - the difference is entirely whether the trace routes into the column algorithm or into free-form reasoning. If the trace does not show columns, do not trust the number.

Reasoning-trace faithfulness

Over 150 straight-arithmetic generations, every A op B = C statement in the trace was checked for arithmetic truth and cross-tabulated against whether the final answer was right:

	first release	this revision
faithful (valid trace → right answer)	97.3%	95.3%
broken trace → wrong answer	2.7%	4.0%
readout (valid trace → wrong answer)	0%	0.7%

The readout cell is the interesting one: it counts cases where the model had a correct computation written in front of it and still produced a different answer. At 0.7% (1 case in 150) the trace is doing the work almost all of the time - which is not the usual finding at this scale. The single failure was a sign misfire on an ordered subtraction, the same mechanism behind the subtraction cells above.

Practical consequence: the working is worth reading. If the columns are right, the answer is almost certainly right.

Training

Ten rounds of arithmetic SFT, each diagnosing a specific failure in the last. The two findings that did the work, both of which are about data, not scale:

1. Control the joint (operation × operand-length) distribution, not the marginals. Round 7 was balanced per-operation but chose lengths independently, so 2-digit problems were 2.1× more likely to be multiplication than addition. The model learned that "2 digits" means use the partial-product template and fired it on addition — 2-digit addition collapsed to 46.3% while 3-digit held 87.7%. Specifying every (op, length) cell explicitly fixed it.

2. Write the padding step down. Subtraction with ragged operands (90210 − 4877) failed until the scratchpad said Pad 4877 to 04877. out loud. Making the alignment an explicit token the model emits, rather than something it had to do silently, took 5-digit subtraction from 70% to 96%.

Two LoRA adapters (r=16, α=32, 2.02% trainable) were then merged in: one adding word-problem and reasoning data, one adding phrasing variety. LoRA was used specifically so the arithmetic routine could not be traded away for the new capability — and it worked better than intended: the word-problem adapter was expected to leave arithmetic flat, but it lifted the average from 94.0 to 99.9, repairing the exact four cells that ten full rounds kept trading against each other. The plausible reason is the constraint itself — at 2% trainable parameters it cannot reallocate capacity between operations, so it can only sharpen the routine already there.

Full recipe

Ten full SFT rounds, a continuous chain from the instruct checkpoint (step 31,196) to step 71,196 — 40,000 steps. Weights are percentages of the sampled mix.

Round	Steps	LR	Data mix
math2	+1,000	2e-5	reasoning-v2 40 · math-v3 26.7 · qa-distill 12.5 · smoltalk 12.5 · longdef-sft 7.5
math3	+4,000	2e-5	same mix, extended
math4	+2,000	1e-5	math-v3 30 · math-v3fix 25 · math-v2 15 · reasoning-v2 15 · smoltalk 10 · qa-distill 5
math5	+2,000	1e-5	same as math4
math6	+3,000	1.5e-5	math-short 60 · math-v3 12 · math-v3fix 10 · smoltalk 10 · qa-distill 5 · reasoning-v2 3
math7	+5,000	1.5e-5	math-short2 55 · math-short 20 · math-v3 8 · math-v3fix 7 · smoltalk 7 · qa-distill 3
math8	+5,000	1.5e-5	math-balanced 65 · math-short2 10 · math-v3 8 · math-v3fix 7 · smoltalk 7 · qa-distill 3
math9	+5,000	1.5e-5	math-balanced2 70 · math-balanced 8 · math-v3 7 · math-v3fix 6 · smoltalk 6 · qa-distill 3
math10	+5,000	1.5e-5	math-balanced3 72 · math-balanced2 8 · math-v3 6 · math-v3fix 5 · smoltalk 6 · qa-distill 3
math11	+8,000	1.5e-5	math-balanced4 80 · smoltalk 8 · math-v3 5 · math-v3fix 4 · qa-distill 3

Then two merged adapters, 1 epoch each:

Adapter	Rows	Composition	LR
word-problem	71,000	80.3% <think> synthetic word problems · 15.3% column scratchpad · 4.4% reasoning prose	1e-4
phrasing	84,000	55.0% column scratchpad (30 question templates) · 28.6% <think> word problems · 16.4% reasoning prose	8e-5

The math-balanced* generators are the ones that mattered; each fixed a specific measured defect in its predecessor.

What each dataset contains

All arithmetic sets are generated locally by deterministic scripts with fixed seeds, so every label is exact by construction rather than model-generated. Names are opaque on their own, so:

Dataset	Contents	The defect it addressed
math-v3, math-v3fix	column scratchpad in ones:/tens:/hundreds: form	the original format; fix corrects phantom place-value columns
math-short, math-short2	shorter arithmetic traces	the 320M recipe's traces were too long to learn from at this scale
math-balanced	explicit (operation x operand-length) joint distribution	lengths sampled independently let multiplication dominate the 2-digit band and steal addition's template
math-balanced2	+ operation cue, worked partial-product sums	the model asserted partial-product sums it could not do mentally
math-balanced3	+ equal-width subtraction	one extra phantom column on same-width problems
math-balanced4	+ explicit Pad 9 to 009. statements	nothing distinguished real zero-padding from a column that should not exist
math-phrased	30 question templates per operation	the model only recognised "What is X plus Y?"
wp-calc	word problems whose <think> block runs the column routine	traces asserted mental results instead of computing
wp-cue	one-step problems that name the linguistic cue before choosing an operation	"drops in 836 more" was being read as subtraction
math-negative	subtraction with negative results, explicit sign decision	every generator ordered its operands, so negatives were never trained or tested
math-mult3	3-digit multipliers	the corpus contained zero examples with a multiplier wider than 2 digits

Ballast, unchanged throughout: smoltalk and qa-distill (general instruction data, 9-11% of every round) prevent the model degenerating into a calculator that cannot hold a sentence. reasoning-v2, gsm8k-cot and reasoning-math are word-problem and chain-of-thought sets; the last two are ~100% <think>-formatted, which is where that behaviour comes from.

The generators are deterministic - same seed, same bytes - so the exact training sets are reconstructible from the scripts without needing the data itself.

Three notes for anyone reproducing this

• The predecessor's recipe did not transfer. Rounds math2/math3 ran the exact data mix that produced the 320M V2 Math (91.5% average). On this model it scored 58.5%. The 350M base scores 0–3% on arithmetic before any math SFT, where the 320M's pre-math model was already at 55–69% — a pretraining difference, not an SFT one. Everything from math6 onward exists because a proven recipe was reused first and measured to fail.
• Straight arithmetic was bought with word problems, then bought back. reasoning-v2 went 40% → 15% → 3% → absent while chasing arithmetic defects, and word-problem accuracy fell to roughly 1 in 4. The first LoRA exists to refill that hole without re-opening the arithmetic trade.
• General-purpose ballast never left. smoltalk + qa-distill held 9–11% in every round. Without it the model degenerates into a calculator that cannot hold a sentence.

Full write-up: [PROJECT_BRIEF.md in the training repo].

Limitations

• Addition degrades past 14 digits — 65% at 16, 25% at 18. These are arithmetic slips in a long carry chain, no longer the silent skipped-column failure of the first release.
• Multiplication is reliable through 3×3 (98.0%). 4×4 is a hard wall. No division at all.
• Negative results degrade with width — 100% at 1 digit, 58% at 5. The magnitude comparison is the weak step, not the arithmetic. The failures are near misses (375 × 903 → 339625 vs 338625), so it is running the partial-product routine and losing digits summing it, not failing to attempt. No division.
• Word problems are weak (GSM8K 4.0%, best word-problem set ASDiv 16.5%). It converts one sentence into one operation sometimes; it does not chain operations.
• Greedy decoding matters. Sampling corrupts the column routine.
• Phrasing-sensitive. It is strongest on the direct "What is X plus Y?" form.
• Not a general assistant — for conversation and facts, use the Instruct variant.
• Trained almost entirely on English.

Related

• TinyBrainBot 350M V3 Base — the pretrained model.
• TinyBrainBot 350M V3 Instruct — general question answering.