Quark-150M-128K-Base

A ~165M-parameter base language model with a 128K-token (131,072) context.

Research artifact. This model is released for research on small language models: architecture, training dynamics, long context and reinforcement learning from a base model. It is not a product and has not been evaluated for production use. It has had no safety training. Like any small language model, it makes factual and reasoning errors and can produce biased or harmful text. Do not use it for decisions that affect people, or deploy it without your own evaluation and safeguards.

Parameters 165.3M (140.0M without embeddings)
Layers 16 (12 local, 4 global)
Context 131,072 tokens
Training tokens ~506B
Vocabulary 32,772 (embedding table padded to 32,896)
Weights float32 safetensors

What this checkpoint is

This is the 128K-context base model. It continues a late checkpoint of the Quark-150M 32K anneal with 10B tokens at 131,072-token sequences. The RoPE base of the global layers was raised from 500K to 4M for the longer context, and four tool-call tokens were added to the vocabulary. It has not been instruction-tuned, but its long-context mix includes chat-formatted and tool-use data.

The Quark-150M family

Model What it is
Quark-150M-32K-Raw Pre-anneal checkpoint: end of the constant learning-rate phase (~472B tokens, trained at up to 4K context)
Quark-150M-32K-Base Base model after the 32K-context anneal (~500B tokens)
Quark-150M-128K-Base Base model extended to 128K context (+10B tokens)
Quark-150M-128K-Zero Reinforcement learning (GRPO) straight from the 128K base, without supervised fine-tuning

Architecture

  • Decoder-only transformer hybrid, 16 layers, width 768, 12 attention heads (head size 64), tied input/output embeddings.
  • Local layers (12): sliding-window attention over the last 256 tokens, with RoPE, QK-norm and a learned attention sink per head. A double-gated short causal convolution (kernel 4) runs in parallel, and the two are blended per channel.
  • Global layers (4, every fourth layer): full causal grouped-query attention (12 query heads, 4 key/value heads) with RoPE and QK-norm.
  • Gated value residuals: every attention after the first layer mixes its values with the first layer's values, through a per-token, per-head gate.
  • Residual stream: depth-scaled residual updates with an RMSNorm before and after every sublayer (sandwich norm). The stream is kept in float32 even when the weights are in bfloat16.
  • MLP: gated MLP with a squared-ReLU gate, hidden size 2048.
  • Small inference state: only the 4 global layers keep a full key/value cache. Local layers keep 255 tokens of keys/values and a 3-token convolution state.
  • Tokenizer: byte-level BPE with 32,768 tokens; every chat tag is a single token.

Training

  • Starting point: a checkpoint late in the 32K-context anneal of Quark-150M (about 496B tokens).

  • Context extension (10B tokens, 131,072-token sequences):

    • books (PG-19, DOAB);
    • textbooks;
    • long PDF documents (FinePDFs, including other languages);
    • arXiv papers;
    • long-context instruction following;
    • synthetic long-context and tool-use sessions;
    • replay of the 32K-stage data.

    Peak learning rate 3e-4, decayed to zero; batches of 1M tokens.

  • Tokenizer: the 32,768-token tokenizer plus [TOOL_CALL], [/TOOL_CALL], [TOOL_RESULT] and [/TOOL_RESULT]. The embedding table is padded to 32,896 rows.

Usage

Requires transformers>=5.23 (tested with 5.23.0) and trust_remote_code=True (the modeling code ships with the weights).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "empero-ai/Quark-150M-128K-Base"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to("cuda")

inputs = tok("Photosynthesis is the process by which", return_tensors="pt").to("cuda")   # prepends <|endoftext|>
out = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.7, top_p=0.9,
                     repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))

Small base models repeat themselves under greedy decoding; sampling with a mild repetition penalty (as above) reads better.

The weights are stored in float32. dtype=torch.bfloat16 halves the memory, and the residual stream still runs in float32. Batched generation works with the tokenizer's default left padding.

Chat format

Turns are wrapped in tags, and each tag is a single special token:

<|endoftext|>[SYSTEM]
You are quark, a helpful assistant made by Empero AI
[/SYSTEM]
[USER]
What is 2 + 3?
[/USER]
[MONOLOGUE]
Two plus three is five.
[/MONOLOGUE]
[ASSISTANT]
5
[/ASSISTANT]

[SYSTEM] and [MONOLOGUE] (the model's private thinking) are optional. <|endoftext|> marks the start of a document; the tokenizer and the chat template add it.

The 128K models also have tool-call tokens. Tools are offered in the system turn, and the template writes them for you from tools=[...]. The model calls a tool with [TOOL_CALL] {"name": ..., "arguments": {...}} [/TOOL_CALL]. The caller runs the tool and appends the result as a [TOOL_RESULT] ... [/TOOL_RESULT] turn (role tool_result, or tool), then the model continues.

Sample completions

Generated with the code above in float32, one run per prompt with the settings shown: nothing was re-rolled or selected. Generation stops at <|endoftext|>, [/ASSISTANT] or [/TOOL_CALL], or at the token limit. Special tokens are shown as text; the prompt is everything up to where the model's output begins.

Text completion (sampled: temperature 0.7, top-p 0.9, repetition penalty 1.1, seed 1234, up to 80 new tokens)

Prompt:

The capital of France is

Output:

 Paris. The capital city of the Kingdom of France is also called Paris, and is located in the center of Europe, which is divided into two regions: the French Republic, and the Kingdom of France itself. It is a very ancient city that has been in existence for more than 2,000 years.
In the beginning, the French Republic was a small kingdom in central France. At the time

Text completion (sampled: temperature 0.7, top-p 0.9, repetition penalty 1.1, seed 1234, up to 80 new tokens)

Prompt:

Photosynthesis is the process by which

Output:

 plants make their own energy through a series of chemical reactions that are similar to those occurring in photosynthesis. Photosynthesizers, also known as green plants, use light, water and carbon dioxide (CO2) to create their own energy from sunlight.
Photophoresis refers to the process by which light triggers photosynthetic processes in plants. The process occurs when a plant absorbs

Text completion (sampled: temperature 0.7, top-p 0.9, repetition penalty 1.1, seed 1234, up to 80 new tokens)

Prompt:

Question: Tom buys 3 apples at $2 each and pays with a $10 bill. How much change does he get?
Answer:

Output:

 Tom starts the shopping with 3 apples and pays with a $10 bill. Then he buys 2 more apples, pays again with a $10 bill, and finally pays with a $10 bill. Adding all these up gives him a total of 6 dollars. What is the change in Tom's wallet?
So, the answer is 7 dollars.
- I found an answer but didn't understand how

Text completion (sampled: temperature 0.7, top-p 0.9, repetition penalty 1.1, seed 1234, up to 80 new tokens)

Prompt:

def is_prime(n):

Output:

    if n == 1:
        return True
    if n < 2:
        return False
    while n > 1:
        if any(p is prime for p in int(n)**2-1):
            return False
    return True

def main():
    if len(input()) < 2:
        print("Please enter a number between 1

Chat (greedy, up to 80 new tokens)

Prompt:

[USER]
What is the boiling point of water at sea level?
[/USER]
[ASSISTANT]

Output:

The boiling point of water at sea level is 4Β°C.
[/ASSISTANT]

Limitations and intended use

  • A ~165M-parameter model: little world knowledge, frequent factual and reasoning errors, and confident hallucinations.
  • Trained mainly on English. Other languages appear in the pretraining data but are weaker.
  • No safety or alignment training. Outputs can be wrong, biased or inappropriate.
  • Released as-is for research. Expect rough edges, and evaluate it on your own task before relying on anything it says.
  • A base model: it continues text and only loosely follows the chat format.

License

The weights and code are released under the Apache License 2.0.


Compute for this work was partly sponsored by lium.io.

Downloads last month
70
Safetensors
Model size
0.2B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for empero-ai/Quark-150M-128K-Base

Finetunes
1 model