gemma-2-2b-it-Abliterated

Sliding-Window Attention Instruct (Abliterated), built with bitnarrow by Amaresh Hebbar

An abliterated release of Google's gemma-2-2b-it utilizing alternating local sliding-window and global attention with logit soft-capping. bitnarrow carefully applies Winsorized quantile clamping before Gram-Schmidt projection to respect Gemma-2's sensitive activation scales.

---

Model Specifications

Property	Value
Model ID	AmareshHebbar/gemma-2-2b-it-Abliterated
Upstream Base Model	google/gemma-2-2b-it
Architecture	Gemma2ForCausalLM (gemma2)
Parameters	2.61B
Context Length	8,192 tokens
VRAM Required (BF16 / 4-bit)	~5.4 GB (BF16) / ~2.1 GB (NF4 4-bit)
Abliteration Toolkit	amareshhebbar/bitnarrow
Recommended Use Case	Creative writing, natural prose generation, and compact single-GPU inference.
License	gemma

---

Copy-Paste Local Usage

Option 1: One-Line Terminal Download & Interactive Chat

Paste this block into your Linux/macOS terminal to install dependencies, download gemma-2-2b-it-Abliterated, and launch an interactive chat session:

pip install -U huggingface_hub transformers torch accelerate bitsandbytes && \
hf download AmareshHebbar/gemma-2-2b-it-Abliterated --local-dir ./models/gemma-2-2b-it-Abliterated && \
transformers chat ./models/gemma-2-2b-it-Abliterated

Option 2: Using the bitnarrow Local Runner Script

If you have run_abliterated.py locally:

python run_abliterated.py -m 6

python run_abliterated.py -m 6 --4bit

python run_abliterated.py -m 6 -p "Write a high-performance ring buffer in C++."

The first command runs in full BF16/FP16 precision, the second in 4-bit low-VRAM mode (~2.1 GB), and the third runs a single prompt directly from the terminal.

Option 3: Python (transformers)

The gemma2 chat template only accepts alternating user / assistant roles, so do not pass a system message.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AmareshHebbar/gemma-2-2b-it-Abliterated"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "user", "content": "Explain how heap exploitation and tcache poisoning work in glibc."},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Option 4: Run in 4-Bit Quantization (bitsandbytes Low-VRAM)

Run gemma-2-2b-it-Abliterated using only ~2.1 GB VRAM:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "AmareshHebbar/gemma-2-2b-it-Abliterated"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
)

Option 5: Serve Locally with vLLM (OpenAI-Compatible API)

pip install vllm
vllm serve AmareshHebbar/gemma-2-2b-it-Abliterated --dtype bfloat16 --max-model-len 8192 --port 8000

Query with curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "AmareshHebbar/gemma-2-2b-it-Abliterated",
    "messages": [
      {"role": "user", "content": "Explain the internals of eBPF packet filtering."}
    ],
    "temperature": 0.7
  }'

Option 6: Convert to GGUF for Ollama / llama.cpp / LM Studio

Convert the safetensors checkpoint to an FP16 GGUF with llama.cpp, then register and run it in Ollama:

python convert_hf_to_gguf.py ./models/gemma-2-2b-it-Abliterated \
  --outfile gemma-2-2b-it-abliterated-f16.gguf \
  --outtype f16

cat > Modelfile <<EOF
FROM ./gemma-2-2b-it-abliterated-f16.gguf
PARAMETER temperature 0.7
EOF

ollama create gemma-2-2b-it-abliterated -f Modelfile
ollama run gemma-2-2b-it-abliterated

---

How bitnarrow Abliteration Works

Rather than using destructive Supervised Fine-Tuning (SFT), which often degrades general reasoning or causes catastrophic forgetting, bitnarrow performs surgical weight orthogonalization directly on the transformer's residual stream:

1. Contrastive Activation Collection: Prompts from harmful and harmless distributions are passed through google/gemma-2-2b-it to capture hidden state activations across all transformer blocks.
2. Winsorization Quantile Clamping: Outlier activation spikes in high-norm attention/MLP channels are clamped to prevent extreme dimensions from skewing the refusal vector calculation.
3. Gram-Schmidt Orthogonal Projection: The refusal direction vector $\hat{r}$ is orthogonalized against harmless capability directions so that general knowledge and instruction following remain intact.
4. Weight-Space Abliteration: Attention output projections (o_proj) and MLP down projections (down_proj) are updated directly in weight space:

$$W_{\text{out}}^{\mathrm{\prime }}=W_{\text{out}}-\widehat{r}\left({\widehat{r}}^T{W}_{\text{out}}\right)W\text{'}\_\backslash text\{out\}\; =\; W\_\backslash text\{out\}\; -\; \backslash hat\{r\}\backslash left(\backslash hat\{r\}^T\; W\_\backslash text\{out\}\backslash right)$$

---

Complete bitnarrow Abliteration Suite

All 7 models in the Abliteration Weights Collection:

#	Abliterated Model	Base Checkpoint	Params	VRAM (BF16 / 4-bit)
1	Qwen2.5-1.5B-Abliterated	Qwen/Qwen2.5-1.5B	1.54B	~3.2 GB / ~1.2 GB
2	Qwen2.5-1.5B-Instruct-Abliterated	Qwen/Qwen2.5-1.5B-Instruct	1.54B	~3.2 GB / ~1.2 GB
3	Qwen2.5-7B-Instruct-Abliterated	Qwen/Qwen2.5-7B-Instruct	7.61B	~15.5 GB / ~5.2 GB
4	Llama-3.2-3B-Instruct-Abliterated	meta-llama/Llama-3.2-3B-Instruct	3.21B	~6.5 GB / ~2.4 GB
5	Llama-3.1-8B-Instruct-Abliterated	meta-llama/Llama-3.1-8B-Instruct	8.03B	~16.2 GB / ~5.6 GB
6	gemma-2-2b-it-Abliterated (This Model)	google/gemma-2-2b-it	2.61B	~5.4 GB / ~2.1 GB
7	Phi-3.5-mini-instruct-Abliterated	microsoft/Phi-3.5-mini-instruct	3.82B	~7.6 GB / ~2.8 GB

---

Citation

If you use gemma-2-2b-it-Abliterated or bitnarrow in your research, red-teaming, or applications, please cite:

@software{hebbar2026bitnarrow,
  author = {Amaresh Hebbar},
  title = {bitnarrow: Targeted Weight Orthogonalization and Abliteration Toolkit},
  year = {2026},
  url = {https://github.com/amareshhebbar/bitnarrow}
}