llaa33219 commited on
Commit
df2381d
Β·
verified Β·
1 Parent(s): 2b283fd

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -27,7 +27,7 @@ metrics:
27
 
28
  <img src="https://raw.githubusercontent.com/llaa33219/MicroMixer-4/refs/heads/main/logo-microt.svg" width="300" alt="MicroT-test1 Logo"/>
29
 
30
- # MicroT-test1-1M
31
 
32
  <img src="https://img.shields.io/badge/Parameters-996%2C736-blue?style=for-the-badge&logo=python&logoColor=white&color=%23007BFF" alt="Parameters"/>
33
  <img src="https://img.shields.io/badge/Architecture-Transformer-orange?style=for-the-badge&color=%23FF6600" alt="Architecture"/>
@@ -55,7 +55,7 @@ metrics:
55
 
56
  ## πŸ“‹ Overview
57
 
58
- **MicroT-test1-1M** is a **996,736-parameter** vanilla decoder-only **transformer** β€” multi-head causal self-attention with RoPE β€” pretrained on Discord conversation data and fine-tuned with **FMSP** on 9,012 general-knowledge QA pairs.
59
 
60
  This is the **1M** member of the **MicroT-test1** family: the registered attention-based **reference baseline** of the MicroMixer-4 project. MicroMixer-4's chassis is a pure MLP-Mixer with *no* attention; MicroT-test1 exists so that "Mixer vs Transformer at the same parameter count" is a fair, controlled comparison. It is deliberately boring β€” the standard 2018–2020 transformer recipe parameterized down to the sub-1M regime: no flash attention, no SwiGLU, no ALiBi, no QKNorm, no sliding window, no MQA/GQA, no MoE.
61
 
 
27
 
28
  <img src="https://raw.githubusercontent.com/llaa33219/MicroMixer-4/refs/heads/main/logo-microt.svg" width="300" alt="MicroT-test1 Logo"/>
29
 
30
+ # MicroT-test1-1M-Discord-Dialogues
31
 
32
  <img src="https://img.shields.io/badge/Parameters-996%2C736-blue?style=for-the-badge&logo=python&logoColor=white&color=%23007BFF" alt="Parameters"/>
33
  <img src="https://img.shields.io/badge/Architecture-Transformer-orange?style=for-the-badge&color=%23FF6600" alt="Architecture"/>
 
55
 
56
  ## πŸ“‹ Overview
57
 
58
+ **MicroT-test1-1M-Discord-Dialogues** is a **996,736-parameter** vanilla decoder-only **transformer** β€” multi-head causal self-attention with RoPE β€” pretrained on Discord conversation data and fine-tuned with **FMSP** on 9,012 general-knowledge QA pairs.
59
 
60
  This is the **1M** member of the **MicroT-test1** family: the registered attention-based **reference baseline** of the MicroMixer-4 project. MicroMixer-4's chassis is a pure MLP-Mixer with *no* attention; MicroT-test1 exists so that "Mixer vs Transformer at the same parameter count" is a fair, controlled comparison. It is deliberately boring β€” the standard 2018–2020 transformer recipe parameterized down to the sub-1M regime: no flash attention, no SwiGLU, no ALiBi, no QKNorm, no sliding window, no MQA/GQA, no MoE.
61