Visual Jev blue and teal Victory V logo

visual-jev-4b

Accurate and Efficient Decisions from Shared Visual Context
Guanxu Yu · Yuhang Yao

🌐 Website  ·  📄 Paper  ·  💻 Code & quickstart  ·  🤗 visual-jev-8b

One image. Independent questions. Shared compute.

An answer-supervised LoRA adapter for Qwen/Qwen3-VL-4B-Instruct. Encode an image and shared context once, run independent question branches together, and return probabilities over the choices in each request. Scores come from the backbone’s existing LM head; no extra decision head is included.

Small model. Strong decisions.

88.5% macro accuracy · +2.77 pp vs. TypeSafe Jev · +5.06 pp vs. OpenAI Decisions

Visual Jev 4B compared with TypeSafe Jev and OpenAI Decisions on the same held-out questions

Four benchmarks, equal-weight macro averages, November 2026. Visual Jev uses task SFT; API training data are unknown. TypeSafe Jev leads on ECE and TextVQA. NLL follows task-local temperature scaling. Results and protocol ↗

Full comparison and evaluation protocol

Test images only (70% of each benchmark's images, split by a fixed hash of the image id); the same questions and option order for every system; each system gets its own temperature fitted on the same calibration images. TallyQA is the full test build with GQA-training duplicates removed. API probabilities are returned to two decimals and floored at 0.005; a failed request counts as a uniform guess. NLL and ECE after temperature scaling; AUROC of the top probability for correct vs. wrong answers. Macro is the equal-weight mean of the four benchmarks.

System GQA SNLI-VE TextVQA TallyQA Macro NLL (TS) ECE (TS) AUROC
Visual Jev 8B 0.919 0.818 0.979 0.855 0.893 0.285 0.021 0.896
Visual Jev 4B 0.916 0.808 0.975 0.841 0.885 0.309 0.025 0.890
TypeSafe Jev 0.885 0.713 0.982 0.851 0.858 0.361 0.013 0.870
OpenAI Decisions 0.834 0.732 0.955 0.818 0.835 0.440 0.020 0.833

We sent the same test questions, with the options in the same order, to two hosted APIs that also return one probability per option: OpenAI's GPT-6 Luna Decisions (openai/gpt-6-luna-decisions-20261006, through OpenRouter) and TypeSafe Jev (jev-1.14.0, whose image input TypeSafe marks as not yet production quality), queried in November 2026. Every system gets its own temperature, fitted on the same calibration images.

Read it with three caveats:

  • The adapters were trained on GQA and SNLI-VE training data; API training data are unknown. Most of the adapters' lead comes from those two tasks. On the two task families no post-training saw (†), the systems are close, and TypeSafe Jev has the best TextVQA accuracy.
  • Both APIs return probabilities rounded to two decimals; we floor them at 0.005 before taking logs. A request that failed (1.9% of questions for TypeSafe Jev, which rejected some image sizes; 0.3% for GPT-6 Luna, all refusals) counts as a uniform guess rather than being dropped.
  • TypeSafe Jev is not deterministic: sending the identical request again changed its top answer for 0.8% of GQA, 2.7% of SNLI-VE and 10.3% of short-TallyQA questions. GPT-6 Luna returned identical probabilities on every repeat.

Source: checked-in API comparison data.

Try the released model

CUDA or Apple Silicon MPS · Inference only · No training dataset required

git clone https://github.com/guanxuyu-sv/Visual-Jev.git
cd Visual-Jev
python3 -m venv .venv && source .venv/bin/activate
pip install torch torchvision
pip install -r code/requirements.txt
python code/examples/quickstart.py \
    --image assets/figures/demo_bottle.jpg \
    --request-file code/examples/bottle_questions.json --device auto

The runner downloads the backbone and this adapter on first use. --device auto selects CUDA first, then Apple MPS. For CUDA, install a compatible PyTorch build before the remaining dependencies; on Apple Silicon, use the regular macOS build shown above.

The request supplies named questions, shared state, and 2–16 candidate choices per question. Evaluation covers at most eight choices. The runner caches the shared input and returns a choice and probabilities for each question.

Six independent questions sharing one visual encoding

The 4B example includes an incorrect_question option for false premises; it selects that option with probability 0.966 for the dog/person mismatch. This is a probability relative to the supplied choices, not a calibrated evidence score.

Calibrating and acting on probabilities

Raw probabilities are overconfident. Collect logits with --save-logits calib.jsonl, add the correct option index as "label", and fit a temperature and risk threshold on labelled examples from your own task:

python code/examples/calibrate.py fit --scores calib.jsonl --risk 0.05 0.1 --out calibration.json
python code/examples/calibrate.py apply --scores new.jsonl --calibration calibration.json --risk 0.05

Use several hundred calibration questions where possible; thresholds are empirical, not guarantees. Keep the option order fixed. Temperatures transferred between tasks calibrated worse in our evaluation. See the calibration notes and the corrected threshold implementation.

Research details

Original ablation evaluation · short TallyQA subset

Equal-weight macro accuracy over four benchmarks, mean ± spread over three seeds. † marks a benchmark absent from the post-training task mix.

Benchmark Original backbone This adapter n
GQA-Choice 0.880 0.916 ± 0.002 5,447
SNLI-VE 0.629 0.808 ± 0.006 1,368
TextVQA-Choice † 0.974 0.975 ± 0.002 1,742
TallyQA-Choice † 0.340 0.345 ± 0.014 338
Macro 0.706 0.761 ± 0.002

Read the table honestly: the entire gain sits on the two task families represented in training. On the two held-out families the adapter is indistinguishable from the untrained backbone — TextVQA moves by 0.001 and TallyQA by 0.005 against a seed spread of 0.014. If your task resembles relational VQA or visual entailment, this adapter helps; if it resembles counting, it does not, and TallyQA at 0.345 against a six-option chance rate of 0.167 is weak in absolute terms for both models.

Evaluation scope: this original ablation table uses the short, count-balanced TallyQA subset. Its 0.761 macro is not the 0.885 macro above, which uses the full TallyQA test build with GQA-training duplicate images removed. In the short build, 46 of 498 images (9.2%) overlap GQA training images; they did not have higher accuracy. See the published corrections.

Load the adapter and reproduce the readout
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

BASE = "Qwen/Qwen3-VL-4B-Instruct"
ADAPTER = "guanxuyu/visual-jev-4b-answer-sft"   # seed 0 lives at the root

model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16,
                                                    device_map="cuda")
model = PeftModel.from_pretrained(model, ADAPTER).eval()
processor = AutoProcessor.from_pretrained(BASE)

# the other two seeds:
#   PeftModel.from_pretrained(model, ADAPTER, subfolder="seed1")

The scores are candidate-token logits read from the backbone's own LM head at one fixed readout position, not free-form generation. Prompt layout, readout position, and candidate tokenisation determine the result, so a custom prompt measures a different setup. The public code repository includes the prompt construction, inference path, and paper reproduction guide; use those to reproduce the reported procedure.

Two further cautions from our own measurements:

  • Decisions were measured in bfloat16. Path-to-path logit deviations we traced are arithmetic, not modelling: they collapse by four orders of magnitude in float32, and the worst disagreeing cases have a decision margin of 0.000.
  • This adapter was trained on two candidate counts only, K in {2, 4}, while the evaluation above presents K from 2 to 8. That gap matters less here than it would for a typed head: the readout is candidate tokens through the backbone's own LM head, with no fixed slot per option, so there are no slots left without gradient. Measured on the varied-K GQA evaluation it reaches 0.916, against 0.919 for a slot-indexed head trained on the varied-K data itself. Treat wider candidate sets as untested coverage rather than as a known failure.
Training configuration and data
Base Qwen/Qwen3-VL-4B-Instruct, vision tower frozen (exclude_modules: ".*visual.*")
Method LoRA on the language tower, r=16, α=32, dropout 0.05
Targets q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable 33,030,144 parameters
Objective full-vocabulary next-token cross-entropy at the readout position
Steps 3,000, batch size 8, grad checkpointing, lr 1e-4, 100 warmup
Images Native resolution (see correction below)
Data question records derived from GQA and SNLI-VE training splits; GQA Choice items presented K in {2, 4} options
Seeds 0 (root), 1 (seed1/), 2 (seed2/)

No new human annotation was collected. The training records are derived from publicly released benchmarks; no images are redistributed here, and the adapter is a set of low-rank parameter deltas carrying no image data. Using it requires obtaining the backbone under its own licence.

Image-resolution correction: the configured max_pixels = 200,704 (196 visual tokens) was ignored by the Transformers processor used in these runs. Training, evaluation, and timing used native resolution; a 500×500 GQA image becomes 256 visual tokens. The quickstart’s --max-pixels has the same limitation.

Limitations

  • Helps only on task families seen in training (see the table).
  • Counting remains weak in absolute terms.
  • Single-machine study on consumer GPUs; all figures are reported with seed spread and sample sizes.
  • English only.
  • Choice mode normalizes over the options supplied for each question; it has no built-in abstention outside that set. To let it reject a false premise or insufficient evidence, include an option such as incorrect_question or not enough evidence. The quickstart's incorrect_question option catches the dog/person mismatch with 0.966 probability. That score is relative to the supplied choices, not a calibrated measure of image evidence.
  • Separately, our trained evidence-sufficiency output, which detects when relevant image evidence has been removed, reduced macro accuracy by 0.013. That result concerns a separate output objective; it does not rule out request-time reject options.

Citation

@misc{yu2026visualjev,
  title        = {Visual Jev: Accurate and Efficient Decisions from Shared Visual Context},
  author       = {Guanxu Yu and Yuhang Yao},
  year         = {2026},
  eprint       = {2609.25845},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url          = {https://arxiv.org/abs/2609.25845}
}

License and resources

This adapter is released under Apache-2.0. The backbone and source datasets retain their respective terms.

Reproduction guide · Experimental outputs · Brand asset sources

Downloads last month
1,819
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for guanxuyu/visual-jev-4b-answer-sft

Adapter
(257)
this model

Space using guanxuyu/visual-jev-4b-answer-sft 1

Paper for guanxuyu/visual-jev-4b-answer-sft