LLaVA-OneVision-0.5B — LiteRT-LM (on-device Vision-Language Model)

llava-hf/llava-onevision-qwen2-0.5b-ov-hf converted to the LiteRT-LM (.litertlm) format for on-device image+text inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

LLaVA-OneVision-0.5B is a compact vision-language model from the LLaVA team: a SigLIP vision encoder + MLP projector feeding a Qwen2-0.5B language decoder. This 829 MB bundle runs it through LiteRT-LM's fast_vlm multimodal path — give it an image and a question, get a grounded answer, fully offline.

File	LLaVA-OneVision-0.5B.litertlm (~829 MB)
Vision	SigLIP encoder (384×384, 729 patches, no CLS) + MLP projector, int8 → 730 image tokens (729 + an image_newline)
Decoder	Qwen2-0.5B (896-dim, 24 layers), int4 weights (symmetric, blockwise-32 + OCTAV); tied embedding INT8 (externalized)
Compute	integer
Context (KV cache)	2048
Image input	resized to 384×384 (OpenAI-CLIP normalization baked into the vision encoder)
Base model	llava-hf/llava-onevision-qwen2-0.5b-ov-hf

How to use

1. Install the runtime

pip install litert-lm

2. Run it in one command — this downloads the bundle, encodes your image and answers:

litert-lm run --from-huggingface-repo litert-community/LLaVA-OneVision-0.5B LLaVA-OneVision-0.5B.litertlm \
  --attachment photo.jpg \
  --prompt "Describe this image in one sentence."

On the COCO sample image huggingface/documentation-images/coco_sample.png (two tabby cats on a pink blanket, remote controls beside them) this prints:

The image is a digital image of a cat sleeping on a pinkish-red surface.

Drop --prompt for an interactive chat, and pass --attachment more than once for several images. litert-lm serve exposes the same bundle as a local OpenAI-compatible API. The same file runs on macOS, Linux and Windows.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine. These figures cover the text path; the vision encoder runs once per image and is not included.

Device	Backend	Prefill (256)	Decode	TTFT
Apple M4 Max (macOS)	CPU	955 tok/s	113.7 tok/s	0.31 s
Apple M4 Max (macOS)	GPU (Metal)	7030 tok/s	239.3 tok/s	0.04 s

Every desktop backend listed above was checked by actually generating a caption on it, not just by reading the benchmark tool's output.

Accuracy note

Single-image VQA produces coherent, image-grounded answers (CPU-verified; the SigLIP vision tower converts bit-faithfully to the reference, float CPU-parity corr ≈ 1.0).

⚠️ Best for single-image VQA — one image per conversation

Ask about one image per chat. This 0.5B model with 730 image tokens per image becomes unreliable when a second image is added to the same conversation (the answer truncates) — start a new conversation for a different image. Single-image VQA, the primary use case, works well.

Run on iPhone / macOS

Use the LiteRT-LM Swift runtime (swift-litert-lm / the LiteRTDemo sample). Load LLaVA-OneVision-0.5B.litertlm with the vision tower enabled (modalities Modality.textImage / [.vision] — a vision-only bundle, no audio tower), attach a photo, and ask a question.

Run on Android — Google AI Edge Gallery

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

Run this model with image input in the official Google AI Edge Gallery app — no custom app needed (the bundle carries the tokenizer, chat template, and image preprocessing config):

1. Push the bundle onto the phone (or download it there directly from this repo): adb push LLaVA-OneVision-0.5B.litertlm /sdcard/Download/
2. Open the Gallery app, tap the + icon (bottom-right) and pick LLaVA-OneVision-0.5B.litertlm in the file picker.
3. In the Import Model dialog, check "Support image" (required for image input), pick GPU (fast) or CPU, then tap Import.
4. Open the Ask Image task, choose the imported model, attach a photo, and ask.

Tip: ask about one image per conversation (start a new chat for a different image) — this 0.5B model is single-image only.

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official LiteRT-LM CLI — including as a local OpenAI-compatible API server:

pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/LLaVA-OneVision-0.5B LLaVA-OneVision-0.5B.litertlm llava-onevision-0.5b
litert-lm run llava-onevision-0.5b     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Conversion notes

• LiteRT-LM fast_vlm bundle: VISION_ENCODER ([1,384,384,3]→[1,729,1152], SigLIP) + VISION_ADAPTER ([1,729,1152]→[1,730,896], projector + the learned image_newline token) + single-token EMBEDDER
  • PREFILL_DECODE (embeddings-input).
• The vision encoder bakes OpenAI-CLIP normalization + the NCHW transpose into the graph; the single base-resolution (no-anyres) path is used so the image always maps to a fixed 730 soft tokens.
• Decoder exported with externalized (tied) embedder.

License

Apache-2.0 (LLaVA-OneVision + the Qwen2 language component). See the base model card. Converted artifacts are released under the same terms.