laya-browser — laya fine-tuned as a browser-agent decision head (drop-in replacement for TypeSafe Jev)

laya (convaiinnovations/laya) is a non-autoregressive "System 1" decision model: one bidirectional encoder pass answers several typed questions (choice / score / noul) with calibrated probabilities, no text generation. Out of the box it is near chance at browser decisions ("which element should I click for this goal?").

This repo fine-tunes it into the decision head of browser-use/jev-ultrafast, whose /v1/systemone request is exactly laya's predict(state, questions): every step, one ~20 ms forward pass picks the operation (CLICK / TYPE_TEXT / SELECT / PRESS_ENTER / SCROLL / DONE …) and its target element. Everything was trained and evaluated locally on one RTX 4070 Ti SUPER (16 GB), no paid API; a local Qwen3-8B-AWQ only writes the text that TYPE_TEXT types.

One model, v17s/. Earlier checkpoints (v10, v10s, v11s, v14s, v15s) were removed from the main branch; they remain in the commit history. This repo is updated only when a new version is clearly better.

Results (v17s, mmBERT-base 322M, 17–23 ms per step)

evaluation	result
Suite B: 18 tasks on 18 sites that appear in no training source, ×3 runs (code/apps/browser_suite_b.py)	100 % (54/54)
Suite A: the original 16 real-site tasks, ×3 (code/apps/browser_suite.py)	85 % (41/48)
Held-out decisions (6,268: live pages, Mind2Web, NNetNav): operation / target top-1	0.761 / 0.606
Held-out synthetic long-horizon forms (webgym, 10 each, 60-step budget): flight / hotel / shop	2/10 · 3/10 · 9/10

• Suite B is the headline number. At start-up it checks that none of its domains appear among the 682 domains of every training source (results/round3/train_domains.json). Its tasks are mostly one to three steps: navigation, site search, man pages, RFC / package search, a shop's search, page 2, sort by price.
• Suite A (the older suite) still fails books-page2 (the "next" link is several scrolls down) and Google Flights.
• Long multi-field forms are the open problem. The model now plans the right sequence (trip type, origin with its autocomplete suggestion, passengers popup, submit, "Modify search" after a wrong submission) but confuses a date field labelled "Departure" with the origin, and runs out of steps. See "What still fails".

Use

huggingface-cli download cklxx/laya-browser --local-dir laya-browser
cd laya-browser/code && uv sync --extra fast     # pinned uv.lock (Python 3.12, torch 2.11, tilelang 0.1.14)
uv run python verify.py                          # downloads v17s, answers one recorded browser step
uv run python verify.py --fast                   # same through the TileLang fast path

As a TypeSafe replacement for jev-ultrafast (apply code/jev-ultrafast.patch to jev-ultrafast 1231850):

python code/apps/systemone_server.py 8791 /path/to/laya-browser/v17s 60   # 60 = split choices wider than 60 options
# jev-ultrafast: TYPESAFE_BASE_URL=http://127.0.0.1:8791

The checkpoint records laya_fmt (v3) and head_max_len_train (768); the server applies the matching input format. The chunk threshold matters: without it a 160-link page (Hacker News) leaves each option ~4 tokens and the choice becomes a coin toss.

What went into v17s

Harness fixes (code/jev-ultrafast.patch). Half of the original failures were harness bugs, not model errors: a PRESS_ENTER control while a filled text field is focused (arXiv's search overlay has no submit button); elements covered by an unrelated element are not offered, a target covered by its own ancestor is clicked through; observation retries while a page is navigating; a choice that fails 3× or repeats on the same URL is excluded.

Training data (v15s: a clean retrain from the mmBERT-base checkpoint, no suite start page and no DAgger data; v17s continues it for one epoch):

• 421 crawled pages with reverse-generated goals, 700 real DONE states, step-2 negatives;
• Mind2Web train (7.3k steps) plus 5.1k steps re-labelled with planner-style sub-goals by a local Qwen (trajectory-conditioned, Plan-and-Act style);
• 8.9k steps from NNetNav-live;
• 2.3k scripted real-browser trajectories on 206 sites (search → Enter, open an article, scroll to page 2, <select> with intent-style goals, pick an autocomplete suggestion);
• 70k steps from webgym (code/finetune/webgym/): a local synthetic environment of flight / hotel search forms and shop listings with randomized widgets (autocomplete inline or behind a trigger, calendars with month navigation and Done/Apply, radio / segmented / native / custom dropdowns, steppers, popups, cookie banners, forms below the fold). A scripted expert acts through jev's own observations and records each step under the full task and under its current sub-goal, plus a sub-goal DONE verified against page state; 12 % harmless detours teach recovery.

Recipe: laya's RLCD (noisy-logit policy gradient + soft CE); v15s 134k items × 3 epochs (4 h) from the base, v17s one more epoch on 200k items (2 h); post-hoc temperature.

Two fixes in v17s: a <select> option's name is kept when a label is shortened to 50 characters (before, "Please select an option Option 1 Option 2 → Option 2" lost the part that matters: Mind2Web dropdown target 0.39 → 0.74), and synthetic category links are real links that are sometimes the answer (dead distractor links had taught the model to skip sidebars). Sub-goals no longer call the origin "departure".

What still fails

• Long forms: still 2/10 flights and 3/10 hotels on held-out synthetic forms; a date field labelled "Departure" is still confused with the origin sometimes.
• Pages where the target is several screens down behind many distracting links (books-page2).
• Sites that block headless Chromium (DuckDuckGo, Bing, most airline and hotel sites) cannot be used at all.
• jev's DOM reader hides password fields by design, so logins are impossible.

Things that did not help: confidence-gated escalation to Qwen3-8B (worse: the fine-tuned model is the better decider on these pages), a run-time sub-goal planner on top of v15s (20 % vs 33 % on the synthetic forms), torch.compile on variable shapes.

Changelog

version	date	change
v10s	2026-09-21	first usable model: format v3, 17–23 ms per step
v14s	2026-09-26	harness fixes, Enter / scroll / select data, NNetNav; suite B 100 %
v15s	2026-09-27	clean retrain without the DAgger set; long-horizon webgym data and sub-goal labels
v17s	2026-09-27	dropdown option names kept, real category links, more typed-date / recovery data; suite A 85 %, webgym 47 %

Correction: the DAgger set used from v10 to v14s came from running a Qwen teacher on suite A itself (152 of its 331 cases are suite-A goals, often with wrong labels), so the suite-A numbers published for those versions were contaminated. v15s is trained without it; suite B was never affected.

Files

v17s/     the model (laya checkpoint dir: model.safetensors, encoder/, tokenizer/, rl_agent_config.json)
code/     server, suites, finetune pipeline (incl. webgym), TileLang kernels, jev-ultrafast patch, verify.py
results/  suite JSONs and logs behind the numbers above (results/v17s/, results/v15s/, results/round3/)
assets/   demo video

License

Apache-2.0, same as laya. Mind2Web and NNetNav are used under their own licenses for training only.