yannelli's picture
Add hero banner to README
047a67d verified
|
Raw History Blame Contribute Delete
16.3 kB
---
license: other
license_name: lfm1.0
license_link: LICENSE
base_model: LiquidAI/d1-3B
base_model_relation: finetune
datasets:
- yannelli/js-ts-code-review-decisions
language:
- en
- code
library_name: transformers
pipeline_tag: text-classification
tags:
- decision-model
- typed-decisions
- system-one
- code-review
- code-quality
- javascript
- typescript
- lfm2
- liquid
- lora
model-index:
- name: d1-3b-code-review-decisions
results:
- task:
type: text-classification
name: Code review decisions
dataset:
name: JS/TS Code Review Decisions
type: yannelli/js-ts-code-review-decisions
config: decisions
split: test
metrics:
- type: f1
name: Macro F1 @ 0.5
value: 0.917
- type: accuracy
name: Accuracy
value: 0.940
- type: log_loss
name: Log loss
value: 0.172
- type: brier_score
name: Brier score
value: 0.072
- type: ece
name: Expected calibration error
value: 0.017
source:
name: score_decisions.py
url: https://huggingface.co/datasets/yannelli/js-ts-code-review-decisions
---
![d1-3B Code Review Decisions: LiquidAI’s d1-3B, fine-tuned to answer 13 typed questions about a JavaScript or TypeScript snippet in one forward pass.](https://huggingface.co/yannelli/d1-3b-code-review-decisions/resolve/main/assets/d1-3b-code-review-decisions-banner.png)
# d1-3B Code Review Decisions
A decision model that reviews JavaScript and TypeScript code. Give it a snippet and the task the snippet implements, and it answers 13 typed questions with probabilities: which of 10 code review flaws the code has, whether it has any flaw, which flaw matters most, and how mergeable it is. It reads the snippet once and answers all 13 questions as branches of one forward pass, without generating text. Of the four models trained on this dataset it scores highest on every metric.
It is [LiquidAI/d1-3B](https://huggingface.co/LiquidAI/d1-3B) fine-tuned on [JS/TS Code Review Decisions](https://huggingface.co/datasets/yannelli/js-ts-code-review-decisions). On that dataset's test split it scores macro F1 0.917 and accuracy 0.940 over 15,600 decisions.
| | |
|---|---|
| Base model | [LiquidAI/d1-3B](https://huggingface.co/LiquidAI/d1-3B), LFM2.5-VL-3B with Liquid AI's d1 decision readout |
| Parameters | 3.1B (bf16, 6.2 GB) |
| Input | A state with `task`, `language` and `code`, and named questions |
| Output | P(yes) for yes/no questions; a distribution over options for choice and score questions |
| Training data | [JS/TS Code Review Decisions](https://huggingface.co/datasets/yannelli/js-ts-code-review-decisions), `decisions` config, `train` split (9,600 snippets) |
| Test results | Macro F1 0.917 (cutoff 0.5), accuracy 0.940, ECE 0.017 |
| License | LFM Open License v1.0 |
## Questions
The model answers the 13 questions of the dataset's `decisions` config. They are the same for every snippet and are stored in the dataset's `manifest.json`. Each question's `criteria` carry the [flaw definition](https://huggingface.co/datasets/yannelli/js-ts-code-review-decisions#flaw-definitions) the labels follow.
| Question | Type | Asks | Answer |
|---|---|---|---|
| `unclear_naming` | noul | Does this code have unclear naming? | P(yes) |
| `noise_comments` | noul | Does this code have noise comments? | P(yes) |
| `dead_code` | noul | Does this code have dead code? | P(yes) |
| `inconsistent_style` | noul | Does this code have inconsistent style? | P(yes) |
| `over_abstracted` | noul | Does this code have over-abstraction? | P(yes) |
| `too_clever` | noul | Does this code have overly clever code? | P(yes) |
| `duplicated_logic` | noul | Does this code have duplicated logic? | P(yes) |
| `error_handling_problem` | noul | Does this code have an error handling problem? | P(yes) |
| `type_safety_hole` | noul | Does this code have a type safety hole? | P(yes) |
| `unsafe_input` | noul | Does this code have unsafe input handling? | P(yes) |
| `has_flaw` | noul | Does this code have at least one code review flaw? | P(yes) |
| `primary_flaw` | choice | Which code review problem matters most in this code? Choose none if the code is clean. | Distribution over `none` and the 10 flaws |
| `quality` | score | How mergeable is this code as written? | Distribution over 5 levels, 0 "Should not merge." to 4 "Exemplary. Merge as is." |
Quality level `i` is `quality` `i + 1` in the dataset's `default` config.
## Usage
```sh
pip install torch==2.14.1 transformers==5.19.0 huggingface_hub safetensors
```
```python
import json
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
model = AutoModel.from_pretrained("yannelli/d1-3b-code-review-decisions",
trust_remote_code=True, dtype=dtype).to(device)
# The 13 questions the model was trained on, from the dataset's manifest.json
manifest = hf_hub_download("yannelli/js-ts-code-review-decisions", "manifest.json", repo_type="dataset")
questions = json.load(open(manifest))["questions"]
state = {
"task": "Define payroll bank account fields and decode a JSON account record, "
"accepting only a nine-digit routing number and a four-digit account suffix.",
"language": "TypeScript",
"code": """import { z } from 'zod';
const bankAccountSchema = z.object({
routingNumber: z.string().regex(/^\\d{9}$/),
accountSuffix: z.string().regex(/^\\d{4}$/)
});
export type PayrollBankAccount = z.infer<typeof bankAccountSchema>;
export function decodeBankAccount(serialized: string): PayrollBankAccount {
const account_payload = JSON.parse(serialized);
return account_payload;
}""",
}
answers = model.system_one(state, questions)["answers"]
print(json.dumps(answers, indent=1))
```
The example is a test-split snippet the model did not see in training. Its gold labels are `inconsistent_style` (`account_payload` next to camelCase names) and `unsafe_input` (the `JSON.parse` result is returned without the schema defined above it). This model's answers, from the code above on CPU:
| Question | Gold | This model |
|---|---|---|
| `unclear_naming` | no | P(yes) 0.00 |
| `noise_comments` | no | P(yes) 0.00 |
| `dead_code` | no | P(yes) 0.03 |
| `inconsistent_style` | yes | P(yes) 0.99 |
| `over_abstracted` | no | P(yes) 0.00 |
| `too_clever` | no | P(yes) 0.00 |
| `duplicated_logic` | no | P(yes) 0.00 |
| `error_handling_problem` | no | P(yes) 0.01 |
| `type_safety_hole` | no | P(yes) 0.01 |
| `unsafe_input` | yes | P(yes) 0.99 |
| `has_flaw` | yes | P(yes) 1.00 |
| `primary_flaw` | `unsafe_input` | `unsafe_input` (0.98) |
| `quality` | level 0 | level 1 (0.51); expected level 0.62 |
`system_one` returns the System One answer shape. A `noul` answer is P(yes). A `choice` answer has the top `choice`, its `confidence`, and `probabilities` over all options. A `score` answer has `score` (the expected level), `confidence`, `probabilities` over levels, and the `legend`. `model.system_one_batch` answers several snippets in one call. The `quality` answer above, rounded:
```json
{
"type": "score",
"score": 0.623,
"confidence": 0.509,
"probabilities": {
"0": 0.435,
"1": 0.509,
"2": 0.055,
"3": 0.001,
"4": 0.0
},
"legend": {
"0": "Should not merge.",
"1": "Needs significant changes before it can merge.",
"2": "Mergeable with review comments.",
"3": "Mergeable with minor comments.",
"4": "Exemplary. Merge as is."
}
}
```
On Apple Silicon set `device = "mps"` (not benchmarked). On CPU the model runs in float32. For faster CPU inference, or to run under llama.cpp or Ollama, use the [GGUF builds](https://huggingface.co/yannelli/code-review-decisions-GGUF).
The model accepts other questions in the same schema, but it was trained and evaluated only on these 13.
## Results
Results cover the `test` split of the `decisions` config: 300 tasks, 1,200 snippets, and 15,600 decisions, scored with the dataset's `score_decisions.py`. Each snippet was one request with all 13 questions. The four fine-tuned models used the same data and scorer.
| Model | Params | Macro F1 @ 0.5 | Accuracy | Log loss | Brier | ECE | Clean judged clean |
|---|---:|---:|---:|---:|---:|---:|---:|
| **d1-3B (this model)** | 3.1B | **0.917** | **0.940** | **0.172** | **0.072** | **0.017** | **0.950** |
| d1-omni-600M | 587M | 0.771 | 0.882 | 0.321 | 0.127 | 0.028 | 0.827 |
| GLiNER2.5-Decide | 486M | 0.687 | 0.863 | 0.398 | 0.149 | 0.039 | 0.683 |
| NanoJev | 596M | 0.452 | 0.831 | 0.459 | 0.173 | 0.038 | 0.000 |
| d1-3B base model, zero-shot | 3.1B | 0.220 | 0.726 | 0.679 | 0.249 | 0.142 | 0.953 |
| TF-IDF + logistic regression (dataset baseline) | - | 0.502 | 0.830 | 0.449 | 0.172 | 0.030 | 0.307 |
Higher is better for macro F1, accuracy, and clean judged clean; lower is better for log loss, Brier, and ECE. Clean judged clean is the share of the 300 clean test snippets with P(`has_flaw`) below 0.5. The dataset card's [Metrics](https://huggingface.co/datasets/yannelli/js-ts-code-review-decisions#metrics) section defines the other metrics.
95% intervals for this model, from 500 bootstrap resamples of test tasks: macro F1 0.904 to 0.928, accuracy 0.935 to 0.946, log loss 0.158 to 0.189, Brier 0.067 to 0.078. By question type its accuracy is 0.975 on the 11 nouls, 0.812 on `primary_flaw`, and 0.684 on `quality`.
### Per question
| Question | Type | Test positives | Accuracy | F1 @ 0.5 | Log loss | ECE |
|---|---|---:|---:|---:|---:|---:|
| `unclear_naming` | noul | 139 | 0.992 | 0.964 | 0.030 | 0.006 |
| `noise_comments` | noul | 178 | 0.999 | 0.997 | 0.002 | 0.001 |
| `dead_code` | noul | 223 | 0.961 | 0.886 | 0.154 | 0.013 |
| `inconsistent_style` | noul | 147 | 0.977 | 0.903 | 0.079 | 0.009 |
| `over_abstracted` | noul | 145 | 0.988 | 0.952 | 0.038 | 0.009 |
| `too_clever` | noul | 164 | 0.981 | 0.929 | 0.045 | 0.010 |
| `duplicated_logic` | noul | 171 | 0.963 | 0.867 | 0.112 | 0.020 |
| `error_handling_problem` | noul | 140 | 0.988 | 0.946 | 0.037 | 0.009 |
| `type_safety_hole` | noul | 148 | 0.973 | 0.879 | 0.106 | 0.015 |
| `unsafe_input` | noul | 165 | 0.962 | 0.845 | 0.142 | 0.021 |
| `has_flaw` | noul | 900 | 0.946 | 0.963 | 0.146 | 0.024 |
| `primary_flaw` | choice | | 0.812 | | 0.591 | 0.035 |
| `quality` | score | | 0.684 | | 0.760 | 0.051 |
### Calibration
Answers are the model's raw softmax (test ECE 0.017). `temperatures.json` holds one temperature per question, fitted on the `validation` split by log loss. Replacing each answer's distribution p with softmax(log p / T) moves test ECE from 0.017 to 0.015. It never changes the top answer or the order of P(yes) values, so F1 and accuracy stay the same.
### Speed
Median wall-clock time for one snippet with all 13 questions, batch size 1. The PyTorch CPU row covers 52 test snippets, the llama.cpp CPU row the first 20, and the L40S row all 1,200.
| Hardware | Runtime | p50 per snippet |
|---|---|---:|
| One H100 SXM | PyTorch, bf16 | 30.5 ms |
| 16 Neoverse V3 CPU cores | PyTorch, fp32 | 8.9 s |
| 16 Neoverse V3 CPU cores | llama.cpp `llama-server`, Q8_0 GGUF | 6.2 s |
| One L40S | llama.cpp `llama-server`, Q8_0 GGUF | 221 ms |
## Intended use
Flag code review problems in JavaScript and TypeScript while code is written or before it reaches review: in an editor, a pre-commit hook, or CI, as hints for a human reviewer. The probabilities let you pick a cutoff per flag and trade false positives against misses.
## Training
- Data: the `train` split of the `decisions` config, 2,400 tasks and 9,600 snippets (124,800 decisions). The state is the task description, language and code. The training files match the SHA-256 hashes in the dataset's published `manifest.json`.
- The dataset keeps the 4 variants of a task in one split, so no test task was seen in training.
| Setting | Value |
|---|---|
| Method | LoRA (r 32, alpha 64, dropout 0.05) on the language model's attention, convolution and MLP projections, merged into the weights after training |
| Objective | Cross entropy over each question's answer tokens; the snippet is the trunk of d1's packed tree and the 13 questions are its branches |
| Schedule | 1 epoch, 1,200 steps of 8 snippets (104 decisions) |
| Optimizer | AdamW, learning rate 2e-4, 5% warmup, linear decay |
| Precision | bf16 |
| Hardware | One H100 SXM, 36 minutes |
| Validation loss | 0.207 at the end of training (300 validation snippets) |
`training/` holds the scripts that trained and scored this model (`liquid_d1.py`, `common.py`, `calibrate.py`). To rerun it:
```sh
pip install peft==0.21.2 pyarrow numpy
hf download yannelli/js-ts-code-review-decisions --repo-type dataset --local-dir js-ts-code-review-decisions
export DECISIONS_DATA=js-ts-code-review-decisions
python training/liquid_d1.py train --model LiquidAI/d1-3B --ckpt ckpt --epochs 1
python training/liquid_d1.py predict --ckpt ckpt --split test --out test-answers.jsonl
python js-ts-code-review-decisions/score_decisions.py test-answers.jsonl --split test
```
## Evaluation files
`eval/test-answers.jsonl` holds this model's answers to all 1,200 test snippets in the scorer's format, and `eval/score.txt` is the scorer's output for it. To rescore:
```sh
python js-ts-code-review-decisions/score_decisions.py eval/test-answers.jsonl --split test
```
## Limitations
- Gold labels come from blind model review and a model audit that accepted 98.3% of findings, not from human reviewers. Scores measure agreement with those labels.
- The snippets are synthetic, 5 to 80 lines long, and 25% of them are clean, so false positive rates on real repositories will differ.
- Training covered only JavaScript, JSX, TypeScript and TSX snippets, each with a one-sentence English `task`; pass one with every snippet.
- Weakest flags by test F1: `unsafe_input` (0.84), `duplicated_logic` (0.87), `type_safety_hole` (0.88). `quality` is the hardest question (accuracy 0.684); 80% of its wrong answers are one level off.
- `unsafe_input` follows the dataset's definition (external data used unchecked); it is not a vulnerability scanner.
## Other models trained on this dataset
| Model | Base | Params | Macro F1 @ 0.5 | Accuracy |
|---|---|---:|---:|---:|
| [d1-3B](https://huggingface.co/yannelli/d1-3b-code-review-decisions) (this model) | LFM2.5-VL-3B with Liquid AI's d1 decision readout | 3.1B | 0.917 | 0.940 |
| [d1-omni-600M](https://huggingface.co/yannelli/d1-omni-600m-code-review-decisions) | LFM2.5 encoder with a decision head | 587M | 0.771 | 0.882 |
| [GLiNER2.5-Decide](https://huggingface.co/yannelli/gliner2.5-decide-code-review-decisions) | GLiNER2 DeBERTa-v3 encoder with a classification head | 486M | 0.687 | 0.863 |
| [NanoJev](https://huggingface.co/yannelli/nanojev-code-review-decisions) | Qwen3-0.6B with a choice head | 596M | 0.452 | 0.831 |
[yannelli/code-review-decisions-GGUF](https://huggingface.co/yannelli/code-review-decisions-GGUF) has GGUF builds of the two Liquid models for llama.cpp and Ollama.
## License
This model is a derivative of [LiquidAI/d1-3B](https://huggingface.co/LiquidAI/d1-3B) and is released under the base model's LFM Open License v1.0, included as `LICENSE`. The license allows commercial use only for entities with less than $10M in annual revenue; read `LICENSE` for the full terms.
The training data, [JS/TS Code Review Decisions](https://huggingface.co/datasets/yannelli/js-ts-code-review-decisions), is CC BY 4.0.
## Citation
```bibtex
@misc{yannelli2026d13bcodereview,
title = {d1-3B Code Review Decisions},
author = {Ryan Yannelli},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/yannelli/d1-3b-code-review-decisions}}
}
@misc{yannelli2026jstscodereviewdecisions,
title = {JS/TS Code Review Decisions},
author = {Ryan Yannelli},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/datasets/yannelli/js-ts-code-review-decisions}}
}
```
## Maintainer
[Ryan Yannelli](https://ryanyannelli.com) | 10/12/2026 | [GitHub: @yannelli](https://github.com/yannelli) | [Hugging Face: yannelli](https://huggingface.co/yannelli)