| --- |
| license: mit |
| datasets: |
| - dleemiller/wiki-sim |
| - sentence-transformers/stsb |
| language: |
| - en |
| metrics: |
| - spearmanr |
| - pearsonr |
| base_model: |
| - answerdotai/ModernBERT-base |
| pipeline_tag: text-classification |
| library_name: sentence-transformers |
| tags: |
| - cross-encoder |
| - modernbert |
| - sts |
| - stsb |
| - stsbenchmark-sts |
| model-index: |
| - name: CrossEncoder based on answerdotai/ModernBERT-base |
| results: |
| - task: |
| type: semantic-similarity |
| name: Semantic Similarity |
| dataset: |
| name: sts test |
| type: sts-test |
| metrics: |
| - type: pearson_cosine |
| value: 0.9162245947821821 |
| name: Pearson Cosine |
| - type: spearman_cosine |
| value: 0.9121555789491528 |
| name: Spearman Cosine |
| - task: |
| type: semantic-similarity |
| name: Semantic Similarity |
| dataset: |
| name: sts dev |
| type: sts-dev |
| metrics: |
| - type: pearson_cosine |
| value: 0.9260833551026787 |
| name: Pearson Cosine |
| - type: spearman_cosine |
| value: 0.9236030687487745 |
| name: Spearman Cosine |
| --- |
| # ModernBERT Cross-Encoder: Semantic Similarity (STS) |
|
|
| Cross encoders are high performing encoder models that compare two texts and output a 0-1 score. |
| I've found the `cross-encoders/roberta-large-stsb` model to be very useful in creating evaluators for LLM outputs. |
| They're simple to use, fast and very accurate. |
|
|
| Like many people, I was excited about the architecture and training uplift from the ModernBERT architecture (`answerdotai/ModernBERT-base`). |
| So I've applied it to the stsb cross encoder, which is a very handy model. Additionally, I've added |
| pretraining from a much larger semi-synthetic dataset `dleemiller/wiki-sim` that targets this kind of objective. |
| The inference performance efficiency, expanded context and simplicity make this a really nice platform as an evaluator model. |
|
|
| --- |
|
|
| ## Features |
| - **High performing:** Achieves **Pearson: 0.9162** and **Spearman: 0.9122** on the STS-Benchmark test set. |
| - **Efficient architecture:** Based on the ModernBERT-base design (149M parameters), offering faster inference speeds. |
| - **Extended context length:** Processes sequences up to 8192 tokens, great for LLM output evals. |
| - **Diversified training:** Pretrained on `dleemiller/wiki-sim` and fine-tuned on `sentence-transformers/stsb`. |
|
|
| --- |
|
|
| ## Performance |
|
|
| | Model | STS-B Test Pearson | STS-B Test Spearman | Context Length | Parameters | Speed | |
| |--------------------------------|--------------------|---------------------|----------------|------------|---------| |
| | `dleemiller/ModernCE-large-sts` | **0.9256** | **0.9215** | **8192** | 395M | **Medium** | |
| | `dleemiller/CrossGemma-sts-300m` | 0.9175 | 0.9135 | 2048 | 303M | **Medium** | |
| | `dleemiller/ModernCE-base-sts` | 0.9162 | 0.9122 | **8192** | 149M | **Fast** | |
| | `cross-encoder/stsb-roberta-large` | 0.9147 | - | 512 | 355M | Slow | |
| | `dleemiller/EttinX-sts-m` | 0.9143 | 0.9102 | **8192** | 149M | **Fast** | |
| | `dleemiller/NeoCE-sts` | 0.9124 | 0.9087 | 4096 | 250M | **Fast** | |
| | `dleemiller/EttinX-sts-s` | 0.9004 | 0.8926 | **8192** | 68M | **Very Fast** | |
| | `cross-encoder/stsb-distilroberta-base` | 0.8792 | - | 512 | 82M | Fast | |
| | `dleemiller/EttinX-sts-xs` | 0.8763 | 0.8689 | **8192** | 32M | **Very Fast** | |
| | `dleemiller/EttinX-sts-xxs` | 0.8414 | 0.8311 | **8192** | 17M | **Very Fast** | |
| | `dleemiller/sts-bert-hash-nano` | 0.7904 | 0.7743 | **8192** | 0.97M | **Very Fast** | |
| | `dleemiller/sts-bert-hash-pico` | 0.7595 | 0.7474 | **8192** | 0.45M | **Very Fast** | |
|
|
|
|
| --- |
|
|
| ## Usage |
|
|
| To use ModernCE for semantic similarity tasks, you can load the model with the Hugging Face `sentence-transformers` library: |
|
|
| ```python |
| from sentence_transformers import CrossEncoder |
| |
| # Load ModernCE model |
| model = CrossEncoder("dleemiller/ModernCE-base-sts") |
| |
| # Predict similarity scores for sentence pairs |
| sentence_pairs = [ |
| ("It's a wonderful day outside.", "It's so sunny today!"), |
| ("It's a wonderful day outside.", "He drove to work earlier."), |
| ] |
| scores = model.predict(sentence_pairs) |
| |
| print(scores) # Outputs: array([0.9184, 0.0123], dtype=float32) |
| ``` |
|
|
| ### Output |
| The model returns similarity scores in the range `[0, 1]`, where higher scores indicate stronger semantic similarity. |
|
|
| --- |
|
|
| ## Training Details |
|
|
| ### Pretraining |
| The model was pretrained on the `pair-score-sampled` subset of the [`dleemiller/wiki-sim`](https://huggingface.co/datasets/dleemiller/wiki-sim) dataset. This dataset provides diverse sentence pairs with semantic similarity scores, helping the model build a robust understanding of relationships between sentences. |
| - **Classifier Dropout:** a somewhat large classifier dropout of 0.3, to reduce overreliance on teacher scores. |
| - **Objective:** STS-B scores from `cross-encoder/stsb-roberta-large`. |
|
|
| ### Fine-Tuning |
| Fine-tuning was performed on the [`sentence-transformers/stsb`](https://huggingface.co/datasets/sentence-transformers/stsb) dataset. |
|
|
| ### Validation Results |
| The model achieved the following test set performance after fine-tuning: |
| - **Pearson Correlation:** 0.9162 |
| - **Spearman Correlation:** 0.9122 |
|
|
| --- |
|
|
| ## Model Card |
|
|
| - **Architecture:** ModernBERT-base |
| - **Tokenizer:** Custom tokenizer trained with modern techniques for long-context handling. |
| - **Pretraining Data:** `dleemiller/wiki-sim (pair-score-sampled)` |
| - **Fine-Tuning Data:** `sentence-transformers/stsb` |
|
|
| --- |
|
|
| ## Thank You |
|
|
| Thanks to the AnswerAI team for providing the ModernBERT models, and the Sentence Transformers team for their leadership in transformer encoder models. |
|
|
| --- |
|
|
| ## Citation |
|
|
| If you use this model in your research, please cite: |
|
|
| ```bibtex |
| @misc{moderncestsb2025, |
| author = {Miller, D. Lee}, |
| title = {ModernCE STS: An STS cross encoder model}, |
| year = {2025}, |
| publisher = {Hugging Face Hub}, |
| url = {https://huggingface.co/dleemiller/ModernCE-base-sts}, |
| } |
| ``` |
|
|
| --- |
|
|
| ## License |
|
|
| This model is licensed under the [MIT License](LICENSE). |
