CIBench Explorer

The entire experiment-learning surface in one place. Browse the 17×7 capability matrix, run any model on any benchmark, explore paper-grade findings, annotate failure modes, see routing decisions, drift, and the full retraction ledger.

⚠️ ANTHROPIC_API_KEY not set — paid (claude-*) manifests will error.

17-model × NoLiMa 4-bin matrix

Each cell links to a stored result_id. Click any row below to load its F1-F8 annotation in the panel beneath the table — no tab-switch required.

Empty cells are intentional, not missing data: cohort carve-outs (Grok 4.20 + gpt-oss-120b held to MRCR-8n only per paper §5) plus the LB-v2 HF dataset drift documented 2026-04-27 (docs/MODEL-CONFIG.md §4 LB-v2 drift). MRCR-SW (524K-1M) was dropped from the figure on 2026-04-27 and now lives in the paper §5.1 narrative.

MODEL-CONFIG.md §0 — NoLiMa table
model	bin	headline	result_id
qwen3.5-122b-a10b-thinking-on	NoLiMa 128K	0.44	a81eca59

model	bin	headline	result_id
deepseek-v3-2	NoLiMa 8K	0.8	a81eca59
deepseek-v3-2	NoLiMa 32K	0.44	765d7c0d
deepseek-v3-2	NoLiMa 64K	0.53	396aac8e
deepseek-v3-2	NoLiMa 128K	0.25	fbf7945f
glm-5-1	NoLiMa 8K	0.88	d375f55c
glm-5-1	NoLiMa 32K	0.31	5f28fd46
glm-5-1	NoLiMa 64K	0.53	323c126f
glm-5-1	NoLiMa 128K	—
kimi-k2.5	NoLiMa 8K	0.9	9c83e4a7
kimi-k2.5	NoLiMa 32K	0.47	82a0e4e1
kimi-k2.5	NoLiMa 64K	0.5	583b8c81
kimi-k2.5	NoLiMa 128K	0.41	8556dbb6
kimi-k2.6	NoLiMa 8K	0.8	00382183
kimi-k2.6	NoLiMa 32K	0.56	4fe1ee84
kimi-k2.6	NoLiMa 64K	0.66	2098657e
kimi-k2.6	NoLiMa 128K	0.41	28869a20
qwen3-6-plus-preview	NoLiMa 8K	0.9	5e2d9c51
qwen3-6-plus-preview	NoLiMa 32K	0.88	d74e8387
qwen3-6-plus-preview	NoLiMa 64K	0.66	e4f8cee2
qwen3-6-plus-preview	NoLiMa 128K	0.53	e83c6e57
gpt-5.5	NoLiMa 8K	0.9	4334c199
gpt-5.5	NoLiMa 32K	0.97	11fac703
gpt-5.5	NoLiMa 64K	0.97	dd7714a7
gpt-5.5	NoLiMa 128K	0.88	577fa00f
deepseek-v4-flash	NoLiMa 8K	0.9	6cf71098
deepseek-v4-flash	NoLiMa 32K	0.88	97ddf9bc
deepseek-v4-flash	NoLiMa 64K	0.91	9192685b
deepseek-v4-flash	NoLiMa 128K	0.75	0fc96787
deepseek-v4-pro	NoLiMa 8K	0.6	b2cc9df4
deepseek-v4-pro	NoLiMa 32K	0.56	a5c0bdfd
deepseek-v4-pro	NoLiMa 64K	0.5	d5d62178
deepseek-v4-pro	NoLiMa 128K	—
qwen3.5-122b-a10b	NoLiMa 8K	0.69	0b028d81
qwen3.5-122b-a10b	NoLiMa 32K	—
qwen3.5-122b-a10b	NoLiMa 64K	0.22	ebecc5ad
qwen3.5-122b-a10b	NoLiMa 128K	0.25	1ca7c5fe
qwen3.5-122b-a10b-thinking-on	NoLiMa 8K	—
qwen3.5-122b-a10b-thinking-on	NoLiMa 32K	0.91	c1efdfaa
qwen3.5-122b-a10b-thinking-on	NoLiMa 64K	—
qwen3.5-122b-a10b-thinking-on	NoLiMa 128K	—
qwen3.5-35b-a3b	NoLiMa 8K	0.38	e108e639
qwen3.5-35b-a3b	NoLiMa 32K	0.34	b6ecd61b
qwen3.5-35b-a3b	NoLiMa 64K	0.1	2787582f
qwen3.5-35b-a3b	NoLiMa 128K	0.09	755215d6
qwen3.5-9b	NoLiMa 8K	0.6	302c5d99
qwen3.5-9b	NoLiMa 32K	0.31	5f8f0ba6
qwen3.5-9b	NoLiMa 64K	0.28	6373783d
qwen3.5-9b	NoLiMa 128K	0.13	d9598a62
gpt-5.4	NoLiMa 8K	1	79b4432a
gpt-5.4	NoLiMa 32K	1
gpt-5.4	NoLiMa 64K	0.7
gpt-5.4	NoLiMa 128K	0.4
grok-4-20	NoLiMa 8K	0.75	400cfd5f
grok-4-20	NoLiMa 32K	—
grok-4-20	NoLiMa 64K	—
grok-4-20	NoLiMa 128K	—
gpt-oss-120b	NoLiMa 8K	0.69	66c502b5
gpt-oss-120b	NoLiMa 32K	—
gpt-oss-120b	NoLiMa 64K	—
gpt-oss-120b	NoLiMa 128K	—
kimi-k2-thinking	NoLiMa 8K	0.94	eabc49ae
kimi-k2-thinking	NoLiMa 32K	0.88	32bade26
kimi-k2-thinking	NoLiMa 64K	0.6	9c48ba15
kimi-k2-thinking	NoLiMa 128K	0.2	0e62a3c3

Selected row — F1-F8 annotation

Click a row above to populate this panel. Uses cibench-store-experiments as the default store.

Run any model on any benchmark

Pick a registered model + a benchmark bin. The result lands in a temp store; the Space hard-caps cost at $1.00/click.

Paper-grade findings backed by stored data

Each finding cites stored result_ids. Click to expand.

Qwen3.5-122B-A10B at NoLiMa 32K scores 0.344 with thinking-OFF (vendor-config artifact) and 0.906 with thinking-ON — a 56-point gap entirely attributable to inference-time reasoning, not the underlying model weights.

config	headline	n_correct	result_id
thinking-OFF (Phase AA)	0.344	11/32	985c3a47
thinking-ON (Phase AB)	0.906	29/32	c1efdfaa
Δ	0.562	+18	(controlled)

docs/MODEL-CONFIG.md §0 + §2 Qwen3.5-122B-A10B section

The 4.7 retrain trades MRCR-8n recall (0.947 → 0.727) for NoLiMa inferential gains (64K: 0.7 → 0.9; 128K: 0.4 → 0.7) — same vendor, same family, same month, opposite direction.

axis	Opus 4.6	Opus 4.7	Δ
MRCR-8n (131K-262K)	0.947	0.727	-0.22
NoLiMa 64K	0.7	0.9	0.2
NoLiMa 128K	0.4	0.7	0.3

paper/cibench-neurips-2026.tex §5.1 + §5.2 + Abstract finding (3)

GPT-5.4 trades MRCR-8n (0.654 → 0.564) for NoLiMa 64K (0.6 → 0.7) — the same shape as Anthropic's flip. Three-vendor pattern, not a vendor-specific quirk.

axis	GPT-5.2	GPT-5.4	Δ
MRCR-8n	0.654	0.564	-0.09
NoLiMa 64K	0.6	0.7	0.1

docs/MODEL-CONFIG.md §6 OpenAI GPT-5.2 / 5.4

K2.6 trades MRCR-8n (0.905 → 0.798) for NoLiMa 32K/64K gains (0.47/0.50 → 0.56/0.66). Three-vendor convergent direction flip across Anthropic, OpenAI, and Moonshot Q1→Q2 2026 retrains.

axis	K2.5	K2.6	Δ
MRCR-8n	0.905	0.798	-0.107
NoLiMa 32K	0.47	0.56	0.09
NoLiMa 64K	0.5	0.66	0.16

docs/MODEL-CONFIG.md §1 + §1.5 (K2.6, K2.5)

Opus 4.7 wide MRCR profile shows 35.7% refusal-rate (10/28 items) at 524K-char haystacks — Anthropic's safety classifier declined to engage with the synthetic filler. Any number below the refusal wall is a floor, not a capability ceiling. F6 fires at ≥25%.

profile	refusal_rate	reliability	F6 fires
Opus 4.7 wide	35.7%	artifact-contaminated	True
Threshold (per F6 wiring MR !111)	≥25%	—	True

MODEL-MEMORY-ONTOLOGY.md §4.0.1; F6 implementation in src/cibench/observatory/failure_annotator.py

OpenAI announced MRCR v2 jumped 36.6% → 74.0% (+37pp) from GPT-5.4 to GPT-5.5. The +37pp magnitude REPLICATES on NoLiMa: +27pp at 64K and +47pp at 128K. Cross-vendor benchmark validation.

bin	GPT-5.4	GPT-5.5	result_id
NoLiMa 8K	—	0.9	4334c199
NoLiMa 32K	1.0	0.969	11fac703
NoLiMa 64K	0.7	0.969	dd7714a7
NoLiMa 128K	0.4	0.875	577fa00f

MR !116; docs/MODEL-CONFIG.md §6 GPT-5.5

V4-Flash (284B/13B-active MoE, $0.14/$0.28 per 1M tokens) measures +10pp at 8K, +44pp at 32K, +38pp at 64K, +50pp at 128K vs V3.2. The cheapest open-weight model in the cohort is now competitive with frontier closed-source on long-context inferential retrieval.

bin	V3.2	V4-Flash	Δ	result_id
NoLiMa 8K	0.8	0.9	0.1	6cf71098
NoLiMa 32K	0.44	0.875	0.44	97ddf9bc
NoLiMa 64K	0.53	0.906	0.38	9192685b
NoLiMa 128K	0.25	0.75	0.5	0fc96787

docs/MODEL-CONFIG.md §4 DeepSeek V4 Pro/Flash (added 2026-04-26)

Annotate any stored result with F1-F8

Paste a result_id (8-char prefix or full UUID) from any store, click Inspect.

ProcurementRouter

Pick a task type + context length. Returns the routing recommendation plus rationale + confidence + fallback.

F7 cross-store drift detection + retraction ledger

Drift = same manifest_hash, different headlines across runs.

Retraction ledger

Source: configs/retractions/v0/retractions.yaml

Retractions
retraction_id	severity	claim	correction
schema-evolution-broke-position-bias-manifest-hash-stability	artifact-contamination	Thinking-on v1 effective_tokens values for 4+ reasoning-default models (GLM-5, Q	Thinking-on v2 measurements with HF_MAX_OUTPUT_TOKENS=8192 produced floors 3-4x higher on the same w

retraction_id	severity	claim	correction
v1-thinking-mode-1024-cap-truncation	truncation	Thinking-on v1 effective_tokens values for 4+ reasoning-default models (GLM-5, Q	Thinking-on v2 measurements with HF_MAX_OUTPUT_TOKENS=8192 produced floors 3-4x higher on the same w
chat-template-kwargs-non-reasoning-regression	provider-integration	`chat_template_kwargs={"enable_thinking": False}` could be sent unconditionally	`chat_template_kwargs` is now only sent when the model prefix matches a reasoning-default family (Qw
effective-tokens-unit-bug	unit-bug	Every effective_tokens value on every profile measured prior to 2026-04-19, as r	Corrected computation is `effective_chars // 4`. All 49 profiles measured before the fix were regene
methodology-mrcr-attribution-overreach	methodology-overreach	CIBench "surfaced the Opus 4.7 long-context regression."	METHODOLOGY.md now explicitly scopes what CIBench CAN and CANNOT claim. Comparative capability claim
recompute-cli-provenance-notes-schema-mismatch	schema-mismatch	All 49 `-fixed` profiles produced by the `cibench ontology recompute` CLI were v	The CLI now writes notes to the top-level `MemoryProfile.notes` list (which does declare the field).
mrcr-v2-hf-adapter-fabricated-schema	schema-mismatch	The `mrcr-v2-hf` corpus adapter loaded the openai/mrcr dataset and yielded Corpu	The adapter now uses the real openai/mrcr schema: prompt is a JSON-encoded list of role/content mess
opus-4-7-3-percent-effective-refusal-wall	artifact-contamination	Claude Opus 4.7 achieved only 3% effective_tokens on the wide position-bias swee	The 3% number is retracted as a capability claim. MODEL-MEMORY- ONTOLOGY.md §4.0.1 documents the ref
opus-4-7-refusal-wall-not-resolved-by-wikipedia-prose	artifact-contamination	MR !70 (opus-4-7-3-percent-effective-refusal-wall) left open the expectation tha	Opus 4.7 cannot currently be measured on wide-sweep simple-NIAH via the CIBench probe. The refusal w
reframed-system-prompt-increased-opus-4-7-refusals	methodology-overreach	docs/experiments/2026-04-20-phase-3-opus-halt-report.md proposed three fixes for	System-prompt benchmark-framing with explicit 'do not refuse' language is contraindicated for Opus 4
schema-evolution-broke-position-bias-manifest-hash-stability	schema-mismatch	CIBench's content-addressed manifest discipline implied that every measurement's	Manifest hash stability is NOT preserved across additive schema changes. The semantic manifest is pr
mrcr-8needle-131k-262k-saturates-qwen3-5-family	methodology-overreach	Phase D's MRCR 8-needle 131K-262K char bin headline numbers implied a ranking ac	Qwen3.5-27B, 35B-A3B, and 122B-A10B are INDISTINGUISHABLE at MRCR 8-needle 131K-262K (all 0.9999). F
kimi-k2-5-chat-template-kwargs-vllm-vs-router-shape	provider-integration	CIBench's HF Inference adapter disabled thinking-mode for reasoning-default mode	K2.5 and K2.6 now send `extra_body: {thinking: {type: "disabled"}}` (the official-API shape) via Nov
gemma-4-hf-repo-case-sensitivity-404	provider-integration	CIBench registered Gemma 4 31B-it at HF repo path `google/gemma-4-31b-it` (lower	Gemma 4 31B-it and 26B-A4B-it now resolve at their correct capital-B paths. Fix is staged; fresh Gem

Stateless replay

Pick a pre-registered public manifest. Run it. The result and its replay command are the same artifact.