Update README.md
Browse files
README.md
CHANGED
|
@@ -27,7 +27,7 @@ metrics:
|
|
| 27 |
|
| 28 |
<img src="https://raw.githubusercontent.com/llaa33219/MicroMixer-4/refs/heads/main/logo-microt.svg" width="300" alt="MicroT-test1 Logo"/>
|
| 29 |
|
| 30 |
-
# MicroT-test1-1M
|
| 31 |
|
| 32 |
<img src="https://img.shields.io/badge/Parameters-996%2C736-blue?style=for-the-badge&logo=python&logoColor=white&color=%23007BFF" alt="Parameters"/>
|
| 33 |
<img src="https://img.shields.io/badge/Architecture-Transformer-orange?style=for-the-badge&color=%23FF6600" alt="Architecture"/>
|
|
@@ -55,7 +55,7 @@ metrics:
|
|
| 55 |
|
| 56 |
## π Overview
|
| 57 |
|
| 58 |
-
**MicroT-test1-1M** is a **996,736-parameter** vanilla decoder-only **transformer** β multi-head causal self-attention with RoPE β pretrained on Discord conversation data and fine-tuned with **FMSP** on 9,012 general-knowledge QA pairs.
|
| 59 |
|
| 60 |
This is the **1M** member of the **MicroT-test1** family: the registered attention-based **reference baseline** of the MicroMixer-4 project. MicroMixer-4's chassis is a pure MLP-Mixer with *no* attention; MicroT-test1 exists so that "Mixer vs Transformer at the same parameter count" is a fair, controlled comparison. It is deliberately boring β the standard 2018β2020 transformer recipe parameterized down to the sub-1M regime: no flash attention, no SwiGLU, no ALiBi, no QKNorm, no sliding window, no MQA/GQA, no MoE.
|
| 61 |
|
|
|
|
| 27 |
|
| 28 |
<img src="https://raw.githubusercontent.com/llaa33219/MicroMixer-4/refs/heads/main/logo-microt.svg" width="300" alt="MicroT-test1 Logo"/>
|
| 29 |
|
| 30 |
+
# MicroT-test1-1M-Discord-Dialogues
|
| 31 |
|
| 32 |
<img src="https://img.shields.io/badge/Parameters-996%2C736-blue?style=for-the-badge&logo=python&logoColor=white&color=%23007BFF" alt="Parameters"/>
|
| 33 |
<img src="https://img.shields.io/badge/Architecture-Transformer-orange?style=for-the-badge&color=%23FF6600" alt="Architecture"/>
|
|
|
|
| 55 |
|
| 56 |
## π Overview
|
| 57 |
|
| 58 |
+
**MicroT-test1-1M-Discord-Dialogues** is a **996,736-parameter** vanilla decoder-only **transformer** β multi-head causal self-attention with RoPE β pretrained on Discord conversation data and fine-tuned with **FMSP** on 9,012 general-knowledge QA pairs.
|
| 59 |
|
| 60 |
This is the **1M** member of the **MicroT-test1** family: the registered attention-based **reference baseline** of the MicroMixer-4 project. MicroMixer-4's chassis is a pure MLP-Mixer with *no* attention; MicroT-test1 exists so that "Mixer vs Transformer at the same parameter count" is a fair, controlled comparison. It is deliberately boring β the standard 2018β2020 transformer recipe parameterized down to the sub-1M regime: no flash attention, no SwiGLU, no ALiBi, no QKNorm, no sliding window, no MQA/GQA, no MoE.
|
| 61 |
|