AlBERTina

AlBERTina is a 140M-parameter Italian encoder model from the AlBERTurin family.

It was trained from scratch on approximately 14B Italian tokens using masked language modeling.

Model Description

AlBERTina is a 140M-parameter encoder-only Transformer model for Italian from the AlBERTurin family.

The model incorporates several architectural improvements over the original BERT architecture, including Pre-RMSNorm, SwiGLU activations, ALiBi positional biases, and a mask-only pre-training objective.

AlBERTina uses:

• 12 Transformer layers
• hidden size of 768
• 12 attention heads
• SwiGLU activations
• Pre-RMSNorm
• ALiBi positional biases
• 1,024-token training sequence length
• 20% mask-only MLM
• Muon optimizer

The model uses gettone, a 32,768-token BPE tokenizer optimized for Italian and shared across the AlBERTurin model family.

The model was trained using Matformer.

AlBERTurin Model Family

Model	Parameters	Training Tokens
AlBERTmini	95M	7B
AlBERTina	140M	14B
AlBERTone101	450M	~101B

Installation

python -m pip install \
  git+https://github.com/mrinaldi97/matformer.git@alberturin-v1

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "AlBERTurin/AlBERTina"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

Citation

If you use AlBERTina in your research, please cite:

Matteo Rinaldi, Marco Madeddu, Calogero Jerik Scozzaro, Matteo Delsanto, Daniele Paolo Radicioni, and Viviana Patti.
AlBERTurin: A Fully Open Family of Italian Encoder Models with Modern Architectures.
CLiC-it 2026.