ada-coder-qwen2.5-1.5b

Ada coder logo: a Jacquard punched card, after Ada Lovelace's loom analogy, whose holes form a teal check mark

Qwen/Qwen2.5-Coder-1.5B-Instruct fine-tuned with LoRA to write Ada 2022 and SPARK code from a task description, spec or signature.

Review every line. The output is not qualified for DO-178C, EN 50128, ISO 26262 or any other certification process. Compile it, test it and, for SPARK, prove it before use.

🚀 Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jgalego/ada-coder-qwen2.5-1.5b"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto")

messages = [
    {"role": "system", "content": "You write Ada 2022 and SPARK code that compiles cleanly with GNAT."},
    {"role": "user", "content": "Write a function that returns the greatest common divisor of two Positive numbers."},
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True))

It also completes raw Ada source with no chat template, which is how it is scored below.

🏋️ Training

Base Qwen/Qwen2.5-Coder-1.5B-Instruct
Data r7_dataset.jsonl from the-clanker-lover/steelman-sft-ada, 6,110 instruction/code pairs that compile with GNAT
Method LoRA, r=16, alpha=32, all linear layers, merged
Steps 756 (2.0 epochs), batch size 16, max length 2048
Learning rate 0.0002, cosine
Mean train loss 0.2271
Hardware NVIDIA A10G, 94 min

Then GRPO on the MBPP-Ada problems in MultiPL-E (mbpp-adb), which share none with HumanEval-Ada. Each step samples 8 completions per problem from the raw prompt at temperature 0.8; GNAT builds and runs each against the problem's tests. Reward: 1 if it passes, 0.25 if it only compiles, 0 otherwise. LoRA r=16 on top of the previous model, merged after each round.

Round Steps Problems per step Learning rate Mean reward, first → last step Hardware
1 45 8 2e-05 0.293 → 0.418 NVIDIA A10G, 39 min

📊 Results

nuprl/MultiPL-E HumanEval-Ada (humaneval-adb, 157 problems). Greedy decoding from the raw prompt, then built and run with the GNAT flags from MultiPL-E's eval_adb.py. A program passes when every pragma Assert holds.

Model n pass@1 Compile rate Runtime errors Timeouts
jgalego/ada-coder-qwen2.5-1.5b 157 0.223 0.682 71 1
jgalego/ada-coder-qwen2.5-1.5b@sft 157 0.204 0.592 61 0
Qwen/Qwen2.5-Coder-1.5B-Instruct 157 0.089 0.35 41 0
jgalego/ada-coder-qwen2.5-1.5b@grpo-2 157 0.204 0.694 77 0

Reference scores reported by the Steelman authors (model card, report) on the same benchmark with their own harness, temperature 0:

Model pass@1 Compile rate
Steelman 14B v0.3 (Qwen2.5-Coder-14B + R7 data) 0.459 0.854
Steelman 14B v0.2 (Qwen2.5-Coder-14B-Instruct + R6 data) 0.471 0.752
Qwen2.5-Coder-14B, untuned 0.344 0.510

⚠️ Limitations

  • Scored on small HumanEval-style functions, not on full programs or packages.
  • Contracts are at most checked at run time (-gnata); nothing here runs GNATprove.
  • Trained on a single dataset; style follows that data.
Downloads last month
338
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for jgalego/ada-coder-qwen2.5-1.5b

Finetuned
(227)
this model
Quantizations
1 model

Datasets used to train jgalego/ada-coder-qwen2.5-1.5b

Collection including jgalego/ada-coder-qwen2.5-1.5b

Evaluation results