TinyBrainBot-3M-v2-Base

A 2.94M-parameter general-purpose base model trained from scratch on 6.05B tokens, as a plain Llama (LlamaForCausalLM): no custom code. It is the successor of TinyBrainBot-3M-Base, rebuilt to be balanced: commonsense, facts and Q&A first, with grade-school arithmetic that holds up on wordings it never saw, instead of a high ArithMark score.

Scores vs other models of 3M parameters or fewer

Zero-shot, full test sets, acc_norm. The (raw acc) row shows plain accuracy. The Intelligence Index is the Open SLM Leaderboard's; the other models' numbers come from that leaderboard.

Model	Params	Int Index	HellaSwag	ARC-E	ARC-C	PIQA	ArithMark-3
TinyBrainBot-3M-v2-Base (this model)	2.94M	10.38	27.29	33.38	22.53	56.75	45.10
(raw acc)		10.61	26.83	33.67	18.69	58.65	45.40
GPT-S3-3M	2.97M	9.61	27.68	34.76	23.12	57.02	39.10
Tokle-3M	2.91M	8.91	27.20	34.85	23.98	55.01	40.80
pulvis-v2	2.96M	8.62	27.93	31.86	22.78	56.86	37.40
Purrence-3M	2.99M	8.49	27.81	34.55	23.63	56.64	34.80

Arithmetic it never trained on

The arithmetic share was tuned so ArithMark-3 stays close to how the model does on problems written by other people. None of these sets was used in training (Gemma and Llama were never used to write training data), all acc_norm:

Held-out set	Score
ArithMark-3 (public, for reference)	45.1
Arithmetic stories written by Gemma (544)	34.2
Arithmetic stories written by Llama (1,093)	36.3
23 arithmetic topics ArithMark-3 lacks, new wordings (1,380)	38.8
Unknown positions, distractors, tables, layouts (480)	37.1
Held-out number formats: large / commas / money	40.8 / 46.7 / 45.2
Bare equations from ArithMark-3 ("43 + 19 =")	35.9
ArithMark-2	27.2

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-3m-v2-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-3m-v2-base")
ids = tok("To keep a cut apple from turning brown, you can", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=20, do_sample=False)[0]))

There is also an F16 GGUF (tinybrainbot-3m-v2-base-F16.gguf, 6 MB) for llama.cpp, LM Studio and Jan. A Q8_0 file isn't possible because the hidden size (144) is not a multiple of 32.

This is a base model: it continues text and does not chat. At this size expect short, simple text.

Model

Parameters	2,942,928 total (2.35M non-embedding + 0.59M tied embedding)
Shape	hidden 144, 10 layers, 3 query heads of 48 with 1 KV head (GQA), SwiGLU 416, RMSNorm, RoPE
Vocabulary	4,096-token byte-level BPE trained for this model: digits split one per token, whitespace kept exactly
Context	1,024 tokens

Training

6.05B tokens in three parts: the first 4.72B are TinyBrainBot-3M-Base's own pretraining (its step 36,000), then 0.81B of new pretraining and a 0.51B decay.

New pretraining (0.81B)	Share
Q&A and facts (our distilled plain-text sets)	25%
Arithmetic in many forms (below)	20%
Everyday / how-to / commonsense (Cosmopedia WikiHow + distilled)	15%
Cosmopedia Stanford + OpenStax	13%
TinyStories	12%
FineWeb-Edu	10%
Wikipedia leads	5%

Decay (0.51B)	Share
Q&A and facts	34.3%
Everyday / how-to / commonsense	22%
Cosmopedia Stanford	15%
Wikipedia leads	10%
TinyStories	10%
Arithmetic	8.7%

The arithmetic mixes several sources so the skill is not tied to one wording: drills generated by our own programs (order of operations, negatives, number formats, number words, whitespace variants, tables, unknown positions, distractors; every answer computed in Python), story problems in sentence frames written by several LLM families (Ling, Mistral; the numbers filled in and computed by Python), and Orca-Math word problems. No benchmark training or test data was used.

Recipe: AdamW (betas 0.9/0.95, weight decay 0.1), LR 2.7e-3 held then decayed to 0 (warmup-stable-decay), 131k tokens per step, fp16 mixed precision, 2x V100.

How this checkpoint was chosen. Several decay branches with different arithmetic shares were run from the same checkpoint (step 42,212). A little story-style arithmetic raised ArithMark-3 much faster than any held-out set, so we published the branch whose public ArithMark-3 stays within about 10 points of the held-out sets, not the one with the highest Index (13.93, but ArithMark-3 62.8 against 41.0 held-out). The published weights are that branch's final checkpoint, not a merge.

Evaluation

Float32, lm-eval 0.4.13 (full sets, 0-shot) and the official ArithMark-3 script:

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-3m-v2-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-3m-v2-base --dtype float32

Limitations

At 3M parameters this is a toy-scale model. It knows few facts, loses coherence after a few sentences, and its arithmetic is unreliable outside simple one-step problems. HellaSwag and ARC are close to chance for every model this size, so the gaps between models there are small.