Instructions to use empero-ai/Quark-150M-128K-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use empero-ai/Quark-150M-128K-Base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="empero-ai/Quark-150M-128K-Base", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("empero-ai/Quark-150M-128K-Base", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use empero-ai/Quark-150M-128K-Base with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "empero-ai/Quark-150M-128K-Base" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "empero-ai/Quark-150M-128K-Base", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/empero-ai/Quark-150M-128K-Base
- SGLang
How to use empero-ai/Quark-150M-128K-Base with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "empero-ai/Quark-150M-128K-Base" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "empero-ai/Quark-150M-128K-Base", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "empero-ai/Quark-150M-128K-Base" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "empero-ai/Quark-150M-128K-Base", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use empero-ai/Quark-150M-128K-Base with Docker Model Runner:
docker model run hf.co/empero-ai/Quark-150M-128K-Base
Quark-150M-128K-Base
A ~165M-parameter base language model with a 128K-token (131,072) context.
Research artifact. This model is released for research on small language models: architecture, training dynamics, long context and reinforcement learning from a base model. It is not a product and has not been evaluated for production use. It has had no safety training. Like any small language model, it makes factual and reasoning errors and can produce biased or harmful text. Do not use it for decisions that affect people, or deploy it without your own evaluation and safeguards.
| Parameters | 165.3M (140.0M without embeddings) |
| Layers | 16 (12 local, 4 global) |
| Context | 131,072 tokens |
| Training tokens | ~506B |
| Vocabulary | 32,772 (embedding table padded to 32,896) |
| Weights | float32 safetensors |
What this checkpoint is
This is the 128K-context base model. It continues a late checkpoint of the Quark-150M 32K anneal with 10B tokens at 131,072-token sequences. The RoPE base of the global layers was raised from 500K to 4M for the longer context, and four tool-call tokens were added to the vocabulary. It has not been instruction-tuned, but its long-context mix includes chat-formatted and tool-use data.
The Quark-150M family
| Model | What it is |
|---|---|
| Quark-150M-32K-Raw | Pre-anneal checkpoint: end of the constant learning-rate phase (~472B tokens, trained at up to 4K context) |
| Quark-150M-32K-Base | Base model after the 32K-context anneal (~500B tokens) |
| Quark-150M-128K-Base | Base model extended to 128K context (+10B tokens) |
| Quark-150M-128K-Zero | Reinforcement learning (GRPO) straight from the 128K base, without supervised fine-tuning |
Architecture
- Decoder-only transformer hybrid, 16 layers, width 768, 12 attention heads (head size 64), tied input/output embeddings.
- Local layers (12): sliding-window attention over the last 256 tokens, with RoPE, QK-norm and a learned attention sink per head. A double-gated short causal convolution (kernel 4) runs in parallel, and the two are blended per channel.
- Global layers (4, every fourth layer): full causal grouped-query attention (12 query heads, 4 key/value heads) with RoPE and QK-norm.
- Gated value residuals: every attention after the first layer mixes its values with the first layer's values, through a per-token, per-head gate.
- Residual stream: depth-scaled residual updates with an RMSNorm before and after every sublayer (sandwich norm). The stream is kept in float32 even when the weights are in bfloat16.
- MLP: gated MLP with a squared-ReLU gate, hidden size 2048.
- Small inference state: only the 4 global layers keep a full key/value cache. Local layers keep 255 tokens of keys/values and a 3-token convolution state.
- Tokenizer: byte-level BPE with 32,768 tokens; every chat tag is a single token.
Training
Starting point: a checkpoint late in the 32K-context anneal of Quark-150M (about 496B tokens).
Context extension (10B tokens, 131,072-token sequences):
- books (PG-19, DOAB);
- textbooks;
- long PDF documents (FinePDFs, including other languages);
- arXiv papers;
- long-context instruction following;
- synthetic long-context and tool-use sessions;
- replay of the 32K-stage data.
Peak learning rate 3e-4, decayed to zero; batches of 1M tokens.
Tokenizer: the 32,768-token tokenizer plus
[TOOL_CALL],[/TOOL_CALL],[TOOL_RESULT]and[/TOOL_RESULT]. The embedding table is padded to 32,896 rows.
Usage
Requires transformers>=5.23 (tested with 5.23.0) and trust_remote_code=True (the modeling code ships
with the weights).
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "empero-ai/Quark-150M-128K-Base"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to("cuda")
inputs = tok("Photosynthesis is the process by which", return_tensors="pt").to("cuda") # prepends <|endoftext|>
out = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.7, top_p=0.9,
repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))
Small base models repeat themselves under greedy decoding; sampling with a mild repetition penalty (as above) reads better.
The weights are stored in float32. dtype=torch.bfloat16 halves the memory, and the residual stream still runs in
float32. Batched generation works with the tokenizer's default left padding.
Chat format
Turns are wrapped in tags, and each tag is a single special token:
<|endoftext|>[SYSTEM]
You are quark, a helpful assistant made by Empero AI
[/SYSTEM]
[USER]
What is 2 + 3?
[/USER]
[MONOLOGUE]
Two plus three is five.
[/MONOLOGUE]
[ASSISTANT]
5
[/ASSISTANT]
[SYSTEM] and [MONOLOGUE] (the model's private thinking) are optional. <|endoftext|> marks the start of a
document; the tokenizer and the chat template add it.
The 128K models also have tool-call tokens. Tools are offered in the system turn, and the template writes
them for you from tools=[...]. The model calls a tool with
[TOOL_CALL] {"name": ..., "arguments": {...}} [/TOOL_CALL]. The caller runs the tool and appends the result as a
[TOOL_RESULT] ... [/TOOL_RESULT] turn (role tool_result, or tool), then the model continues.
Sample completions
Generated with the code above in float32, one run per prompt with the settings shown: nothing was re-rolled or selected. Generation stops at <|endoftext|>, [/ASSISTANT] or [/TOOL_CALL], or at the token limit. Special tokens are shown as text; the prompt is everything up to where the model's output begins.
Text completion (sampled: temperature 0.7, top-p 0.9, repetition penalty 1.1, seed 1234, up to 80 new tokens)
Prompt:
The capital of France is
Output:
Paris. The capital city of the Kingdom of France is also called Paris, and is located in the center of Europe, which is divided into two regions: the French Republic, and the Kingdom of France itself. It is a very ancient city that has been in existence for more than 2,000 years.
In the beginning, the French Republic was a small kingdom in central France. At the time
Text completion (sampled: temperature 0.7, top-p 0.9, repetition penalty 1.1, seed 1234, up to 80 new tokens)
Prompt:
Photosynthesis is the process by which
Output:
plants make their own energy through a series of chemical reactions that are similar to those occurring in photosynthesis. Photosynthesizers, also known as green plants, use light, water and carbon dioxide (CO2) to create their own energy from sunlight.
Photophoresis refers to the process by which light triggers photosynthetic processes in plants. The process occurs when a plant absorbs
Text completion (sampled: temperature 0.7, top-p 0.9, repetition penalty 1.1, seed 1234, up to 80 new tokens)
Prompt:
Question: Tom buys 3 apples at $2 each and pays with a $10 bill. How much change does he get?
Answer:
Output:
Tom starts the shopping with 3 apples and pays with a $10 bill. Then he buys 2 more apples, pays again with a $10 bill, and finally pays with a $10 bill. Adding all these up gives him a total of 6 dollars. What is the change in Tom's wallet?
So, the answer is 7 dollars.
- I found an answer but didn't understand how
Text completion (sampled: temperature 0.7, top-p 0.9, repetition penalty 1.1, seed 1234, up to 80 new tokens)
Prompt:
def is_prime(n):
Output:
if n == 1:
return True
if n < 2:
return False
while n > 1:
if any(p is prime for p in int(n)**2-1):
return False
return True
def main():
if len(input()) < 2:
print("Please enter a number between 1
Chat (greedy, up to 80 new tokens)
Prompt:
[USER]
What is the boiling point of water at sea level?
[/USER]
[ASSISTANT]
Output:
The boiling point of water at sea level is 4Β°C.
[/ASSISTANT]
Limitations and intended use
- A ~165M-parameter model: little world knowledge, frequent factual and reasoning errors, and confident hallucinations.
- Trained mainly on English. Other languages appear in the pretraining data but are weaker.
- No safety or alignment training. Outputs can be wrong, biased or inappropriate.
- Released as-is for research. Expect rough edges, and evaluate it on your own task before relying on anything it says.
- A base model: it continues text and only loosely follows the chat format.
License
The weights and code are released under the Apache License 2.0.
Compute for this work was partly sponsored by lium.io.
- Downloads last month
- 70