Instructions to use guanxuyu/visual-jev-4b-answer-sft with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use guanxuyu/visual-jev-4b-answer-sft with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-VL-4B-Instruct") model = PeftModel.from_pretrained(base_model, "guanxuyu/visual-jev-4b-answer-sft") - Notebooks
- Google Colab
- Kaggle

visual-jev-4b
Accurate and Efficient Decisions from Shared Visual Context
Guanxu Yu · Yuhang Yao
🌐 Website · 📄 Paper · 💻 Code & quickstart · 🤗 visual-jev-8b
One image. Independent questions. Shared compute.
An answer-supervised LoRA adapter for Qwen/Qwen3-VL-4B-Instruct. Encode an image and shared context once, run independent question branches together, and return probabilities over the choices in each request. Scores come from the backbone’s existing LM head; no extra decision head is included.
Small model. Strong decisions.
88.5% macro accuracy · +2.77 pp vs. TypeSafe Jev · +5.06 pp vs. OpenAI Decisions
Four benchmarks, equal-weight macro averages, November 2026. Visual Jev uses task SFT; API training data are unknown. TypeSafe Jev leads on ECE and TextVQA. NLL follows task-local temperature scaling. Results and protocol ↗
Full comparison and evaluation protocol
Test images only (70% of each benchmark's images, split by a fixed hash of the image id); the same questions and option order for every system; each system gets its own temperature fitted on the same calibration images. TallyQA is the full test build with GQA-training duplicates removed. API probabilities are returned to two decimals and floored at 0.005; a failed request counts as a uniform guess. NLL and ECE after temperature scaling; AUROC of the top probability for correct vs. wrong answers. Macro is the equal-weight mean of the four benchmarks.
| System | GQA | SNLI-VE | TextVQA | TallyQA | Macro | NLL (TS) | ECE (TS) | AUROC |
|---|---|---|---|---|---|---|---|---|
| Visual Jev 8B | 0.919 | 0.818 | 0.979 | 0.855 | 0.893 | 0.285 | 0.021 | 0.896 |
| Visual Jev 4B | 0.916 | 0.808 | 0.975 | 0.841 | 0.885 | 0.309 | 0.025 | 0.890 |
| TypeSafe Jev | 0.885 | 0.713 | 0.982 | 0.851 | 0.858 | 0.361 | 0.013 | 0.870 |
| OpenAI Decisions | 0.834 | 0.732 | 0.955 | 0.818 | 0.835 | 0.440 | 0.020 | 0.833 |
We sent the same test questions, with the options in the same order, to two
hosted APIs that also return one probability per option: OpenAI's
GPT-6 Luna Decisions (openai/gpt-6-luna-decisions-20261006, through
OpenRouter) and TypeSafe Jev (jev-1.14.0, whose image input TypeSafe
marks as not yet production quality), queried in November 2026. Every system
gets its own temperature, fitted on the same calibration images.
Read it with three caveats:
- The adapters were trained on GQA and SNLI-VE training data; API training data are unknown. Most of the adapters' lead comes from those two tasks. On the
two task families no post-training saw (
†), the systems are close, and TypeSafe Jev has the best TextVQA accuracy. - Both APIs return probabilities rounded to two decimals; we floor them at 0.005 before taking logs. A request that failed (1.9% of questions for TypeSafe Jev, which rejected some image sizes; 0.3% for GPT-6 Luna, all refusals) counts as a uniform guess rather than being dropped.
- TypeSafe Jev is not deterministic: sending the identical request again changed its top answer for 0.8% of GQA, 2.7% of SNLI-VE and 10.3% of short-TallyQA questions. GPT-6 Luna returned identical probabilities on every repeat.
Source: checked-in API comparison data.
Try the released model
CUDA or Apple Silicon MPS · Inference only · No training dataset required
git clone https://github.com/guanxuyu-sv/Visual-Jev.git
cd Visual-Jev
python3 -m venv .venv && source .venv/bin/activate
pip install torch torchvision
pip install -r code/requirements.txt
python code/examples/quickstart.py \
--image assets/figures/demo_bottle.jpg \
--request-file code/examples/bottle_questions.json --device auto
The runner downloads the backbone and this adapter on first use. --device auto selects CUDA first, then Apple MPS. For CUDA, install a compatible PyTorch build before the remaining dependencies; on Apple Silicon, use the regular macOS build shown above.
The request supplies named questions, shared state, and 2–16 candidate choices per question. Evaluation covers at most eight choices. The runner caches the shared input and returns a choice and probabilities for each question.
The 4B example includes an incorrect_question option for false premises; it selects that option with probability 0.966 for the dog/person mismatch. This is a probability relative to the supplied choices, not a calibrated evidence score.
Calibrating and acting on probabilities
Raw probabilities are overconfident. Collect logits with --save-logits calib.jsonl, add the correct option index as "label", and fit a temperature and risk threshold on labelled examples from your own task:
python code/examples/calibrate.py fit --scores calib.jsonl --risk 0.05 0.1 --out calibration.json
python code/examples/calibrate.py apply --scores new.jsonl --calibration calibration.json --risk 0.05
Use several hundred calibration questions where possible; thresholds are empirical, not guarantees. Keep the option order fixed. Temperatures transferred between tasks calibrated worse in our evaluation. See the calibration notes and the corrected threshold implementation.
Research details
Original ablation evaluation · short TallyQA subset
Equal-weight macro accuracy over four benchmarks, mean ± spread over three
seeds. † marks a benchmark absent from the post-training task mix.
| Benchmark | Original backbone | This adapter | n |
|---|---|---|---|
| GQA-Choice | 0.880 | 0.916 ± 0.002 | 5,447 |
| SNLI-VE | 0.629 | 0.808 ± 0.006 | 1,368 |
TextVQA-Choice † |
0.974 | 0.975 ± 0.002 | 1,742 |
TallyQA-Choice † |
0.340 | 0.345 ± 0.014 | 338 |
| Macro | 0.706 | 0.761 ± 0.002 |
Read the table honestly: the entire gain sits on the two task families represented in training. On the two held-out families the adapter is indistinguishable from the untrained backbone — TextVQA moves by 0.001 and TallyQA by 0.005 against a seed spread of 0.014. If your task resembles relational VQA or visual entailment, this adapter helps; if it resembles counting, it does not, and TallyQA at 0.345 against a six-option chance rate of 0.167 is weak in absolute terms for both models.
Evaluation scope: this original ablation table uses the short, count-balanced TallyQA subset. Its 0.761 macro is not the 0.885 macro above, which uses the full TallyQA test build with GQA-training duplicate images removed. In the short build, 46 of 498 images (9.2%) overlap GQA training images; they did not have higher accuracy. See the published corrections.
Load the adapter and reproduce the readout
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
BASE = "Qwen/Qwen3-VL-4B-Instruct"
ADAPTER = "guanxuyu/visual-jev-4b-answer-sft" # seed 0 lives at the root
model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16,
device_map="cuda")
model = PeftModel.from_pretrained(model, ADAPTER).eval()
processor = AutoProcessor.from_pretrained(BASE)
# the other two seeds:
# PeftModel.from_pretrained(model, ADAPTER, subfolder="seed1")
The scores are candidate-token logits read from the backbone's own LM head at one fixed readout position, not free-form generation. Prompt layout, readout position, and candidate tokenisation determine the result, so a custom prompt measures a different setup. The public code repository includes the prompt construction, inference path, and paper reproduction guide; use those to reproduce the reported procedure.
Two further cautions from our own measurements:
- Decisions were measured in
bfloat16. Path-to-path logit deviations we traced are arithmetic, not modelling: they collapse by four orders of magnitude infloat32, and the worst disagreeing cases have a decision margin of 0.000. - This adapter was trained on two candidate counts only, K in {2, 4}, while the evaluation above presents K from 2 to 8. That gap matters less here than it would for a typed head: the readout is candidate tokens through the backbone's own LM head, with no fixed slot per option, so there are no slots left without gradient. Measured on the varied-K GQA evaluation it reaches 0.916, against 0.919 for a slot-indexed head trained on the varied-K data itself. Treat wider candidate sets as untested coverage rather than as a known failure.
Training configuration and data
| Base | Qwen/Qwen3-VL-4B-Instruct, vision tower frozen (exclude_modules: ".*visual.*") |
| Method | LoRA on the language tower, r=16, α=32, dropout 0.05 |
| Targets | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Trainable | 33,030,144 parameters |
| Objective | full-vocabulary next-token cross-entropy at the readout position |
| Steps | 3,000, batch size 8, grad checkpointing, lr 1e-4, 100 warmup |
| Images | Native resolution (see correction below) |
| Data | question records derived from GQA and SNLI-VE training splits; GQA Choice items presented K in {2, 4} options |
| Seeds | 0 (root), 1 (seed1/), 2 (seed2/) |
No new human annotation was collected. The training records are derived from publicly released benchmarks; no images are redistributed here, and the adapter is a set of low-rank parameter deltas carrying no image data. Using it requires obtaining the backbone under its own licence.
Image-resolution correction: the configured max_pixels = 200,704 (196 visual tokens) was ignored by the Transformers processor used in these runs. Training, evaluation, and timing used native resolution; a 500×500 GQA image becomes 256 visual tokens. The quickstart’s --max-pixels has the same limitation.
Limitations
- Helps only on task families seen in training (see the table).
- Counting remains weak in absolute terms.
- Single-machine study on consumer GPUs; all figures are reported with seed spread and sample sizes.
- English only.
- Choice mode normalizes over the options supplied for each question; it has no
built-in abstention outside that set. To let it reject a false premise or
insufficient evidence, include an option such as
incorrect_questionornot enough evidence. The quickstart'sincorrect_questionoption catches the dog/person mismatch with 0.966 probability. That score is relative to the supplied choices, not a calibrated measure of image evidence. - Separately, our trained evidence-sufficiency output, which detects when relevant image evidence has been removed, reduced macro accuracy by 0.013. That result concerns a separate output objective; it does not rule out request-time reject options.
Citation
@misc{yu2026visualjev,
title = {Visual Jev: Accurate and Efficient Decisions from Shared Visual Context},
author = {Guanxu Yu and Yuhang Yao},
year = {2026},
eprint = {2609.25845},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2609.25845}
}
License and resources
This adapter is released under Apache-2.0. The backbone and source datasets retain their respective terms.
Reproduction guide · Experimental outputs · Brand asset sources
- Downloads last month
- 1,819
Model tree for guanxuyu/visual-jev-4b-answer-sft
Base model
Qwen/Qwen3-VL-4B-Instruct
