Single-file CMF build for local inference (Rust; Vulkan, Metal, CPU)
#5
by infosave - opened
We packaged d1-Omni-2B as single CMF files that run with cortiq, a Rust inference engine with no Python: cortiq system-one and /v1/system_one answer named questions over text, JSON, pictures or audio clips from one forward pass, with the release's calibration applied, on Vulkan/DX12, Metal or the CPU.
Files: https://huggingface.co/infosave/d1-Omni-2B-cmf — q8_2f (2.63 GB, 22 of 22 answers as Transformers) and q4tp (1.80 GB, 21 of 22; the one difference is a near tie in the original too).
Three questions about a 3.3 s clip take 0.30 s on an Apple M4 and 0.47 s on an NVIDIA A40.
cargo install cortiq-cli
cortiq system-one d1-omni-2b-q8_2f.cmf --audio call.wav \
--questions '{"people": {"type": "number", "instructions": "For how many people?", "range": [0, 10], "step": 1}}'
(cortiq 0.8.50)