Initial: Harbor Visualiser — Gradio Space for browsing Harbor task spec datasets

Accepts:
owner/name → HF Hub
hf://owner/name[@rev] → HF Hub (explicit)
gh://owner/repo[@ref] → GitHub
https://github.com/owner/repo[.git] → GitHub (full URL)
harbor://name | harbor://org/name[@tag] → Harbor registry (via harbor CLI)
/local/abs/path → local directory

Per task tabs: Overview · Instruction · Patch · test.sh · Dockerfile ·
solve.sh · Raw task.toml. Recognizes Repo2RLEnv's `[metadata.repo2env]`
extension and renders it as a separate metadata table when present.

URL prefill via `?dataset=<uri>` (also `?d=...`) reads on page load.

Live-tested against all four sources before initial push.

.gitignore

@@ -0,0 +1,14 @@
+__pycache__/
+*.py[cod]
+*$py.class
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.venv/
+venv/
+env/
+.env
+.env.local
+.DS_Store
+.harbor-cache/
+*.egg-info/

README.md

@@ -0,0 +1,91 @@
+---
+title: Harbor Visualiser
+emoji: 🔭
+colorFrom: orange
+colorTo: yellow
+sdk: gradio
+sdk_version: "6.14.0"
+app_file: app.py
+pinned: false
+license: apache-2.0
+short_description: Browse Harbor task specs from HF Hub or GitHub in your browser
+---
+
+# Harbor Visualiser
+
+A tiny Gradio Space for browsing [Harbor](https://www.harborframework.com/) task spec directories — the dataset format used by Harbor for agent evaluation + RL environments.
+
+Drop in a Hugging Face dataset id, a GitHub repo, or a local Harbor dataset directory; the viewer renders every task's metadata, instruction, oracle patch, test script, and Dockerfile side-by-side.
+
+## Use it
+
+**Hosted (HF Space):** [https://huggingface.co/spaces/AdithyaSK/harbor-visualiser](https://huggingface.co/spaces/AdithyaSK/harbor-visualiser)
+
+Prefill via URL param:
+```
+https://huggingface.co/spaces/AdithyaSK/harbor-visualiser?dataset=<owner>/<dataset>
+```
+
+**Inputs accepted:**
+
+| Form | Source |
+|---|---|
+| `owner/name` | HF Hub dataset (default) |
+| `hf://owner/name` | HF Hub (explicit) |
+| `hf://owner/name@<rev>` | HF Hub revision pin |
+| `gh://owner/repo` | GitHub repo |
+| `gh://owner/repo@<ref>` | GitHub at branch / tag / SHA |
+| `https://github.com/owner/repo` | Full GitHub URL |
+
+## Run locally
+
+```bash
+pip install -r requirements.txt
+python app.py
+# → http://127.0.0.1:7860
+```
+
+## What it shows per task
+
+| Tab | Source file |
+|---|---|
+| Overview | parsed `task.toml` ([task], [metadata]) + `[metadata.repo2env]` if present |
+| Instruction | `instruction.md` |
+| Patch (oracle) | `solution/patch.diff` |
+| `test.sh` | `tests/test.sh` |
+| Dockerfile | `environment/Dockerfile` |
+| `solve.sh` | `solution/solve.sh` (when present) |
+| Raw `task.toml` | full file |
+
+## Dataset layout it expects (Harbor's standard)
+
+Either of these:
+
+```
+# Layout A — flat (what Repo2RLEnv emits + most git repos use)
+<dataset-root>/
+├── <task-id>/
+│   ├── task.toml
+│   ├── instruction.md
+│   ├── solution/
+│   │   ├── patch.diff
+│   │   └── solve.sh
+│   ├── tests/test.sh
+│   └── environment/Dockerfile
+└── <task-id>/...
+
+# Layout B — nested (what `repo2rlenv push` stages on the Hub)
+<dataset-root>/
+├── registry.json
+├── README.md
+└── tasks/
+    └── <task-id>/
+        └── ... (same as Layout A)
+```
+
+## Stack
+
+- [Gradio 5](https://www.gradio.app/) — UI
+- [huggingface_hub](https://github.com/huggingface/huggingface_hub) — HF dataset download
+- `git` (system binary) — GitHub clone
+- Python stdlib `tomllib` — task.toml parsing

app.py

@@ -0,0 +1,362 @@
+"""Harbor Visualiser — a Gradio Space for browsing Harbor task specs.
+
+Run locally:
+    pip install -r requirements.txt
+    python app.py
+
+Or deploy to a Hugging Face Space — the `README.md` frontmatter pins
+`sdk: gradio` and `app_file: app.py`, so the Space picks this up directly.
+
+URL prefill:
+    https://<space>/?dataset=owner/name
+    https://<space>/?dataset=hf://owner/name@rev
+    https://<space>/?dataset=gh://owner/repo
+    https://<space>/?d=owner/name           (short alias)
+"""
+
+from __future__ import annotations
+
+import logging
+from pathlib import Path
+
+import gradio as gr
+
+from viewer import (
+    DatasetSource,
+    HarborTask,
+    fetch_dataset,
+    list_tasks,
+    load_task,
+    parse_dataset_uri,
+)
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
+logger = logging.getLogger("harbor-visualiser")
+
+
+# ---------------------------------------------------------------------------
+# Backend handlers
+# ---------------------------------------------------------------------------
+
+
+def load_dataset_action(uri: str):
+    """Top-level "Load" button handler.
+
+    Returns a tuple matching the `outputs=` of the button binding:
+        (status_md, source_state, root_state, task_choices, first_task, *task_panel_outputs)
+    """
+    if not uri or not uri.strip():
+        return _empty_state("Enter a dataset URI to begin.")
+
+    try:
+        source = parse_dataset_uri(uri)
+    except ValueError as exc:
+        return _empty_state(f"❌ {exc}")
+
+    try:
+        root = fetch_dataset(source)
+    except Exception as exc:
+        logger.exception("fetch failed")
+        return _empty_state(f"❌ fetch failed: {exc}")
+
+    tasks = list_tasks(root)
+    if not tasks:
+        return _empty_state(
+            f"⚠ No `task.toml` files found in `{source.display}`. "
+            f"Looked under `{root}` for `<id>/task.toml` and `tasks/<id>/task.toml`."
+        )
+
+    first = tasks[0]
+    task = load_task(root, first)
+    status = (
+        f"✅ Loaded **{source.display}** — {len(tasks)} task"
+        f"{'s' if len(tasks) != 1 else ''} found."
+    )
+    return (
+        status,
+        source.display,
+        str(root),
+        gr.update(choices=tasks, value=first, label=f"Tasks ({len(tasks)})"),
+        *_render_task(task),
+    )
+
+
+def select_task_action(task_id: str, root: str):
+    """Radio-button onchange — switch which task's detail tab content is shown."""
+    if not task_id or not root:
+        return _render_task(None)
+    try:
+        task = load_task(Path(root), task_id)
+    except Exception as exc:
+        logger.exception("load_task failed")
+        return _render_task(None, error=str(exc))
+    return _render_task(task)
+
+
+def init_from_url(request: gr.Request):
+    """Read `?dataset=` (or `?d=`) on page load and prefill the input."""
+    if request is None:
+        return ""
+    qs = dict(request.query_params or {})
+    return (qs.get("dataset") or qs.get("d") or "").strip()
+
+
+# ---------------------------------------------------------------------------
+# Rendering helpers
+# ---------------------------------------------------------------------------
+
+
+_EMPTY_OUTPUTS_COUNT = 7  # overview + instruction + patch + test + dockerfile + solve + raw
+
+
+def _empty_state(status: str):
+    """Reset the UI when no dataset is loaded."""
+    return (
+        status,
+        "",  # source_state
+        "",  # root_state
+        gr.update(choices=[], value=None, label="Tasks"),
+        *_render_task(None),
+    )
+
+
+def _render_task(task: HarborTask | None, error: str | None = None):
+    """Compute the 7 panel outputs for a single selected task."""
+    if task is None:
+        msg = error or "Pick a task from the list."
+        return (msg, "", "", "", "", "", "")
+
+    overview_md = _overview_markdown(task)
+    instruction = task.instruction_md or task.instruction_inline or "_(no instruction.md)_"
+    patch = task.oracle_patch or "(no solution/patch.diff)"
+    test = task.test_sh or "(no tests/test.sh)"
+    dockerfile = task.dockerfile or "(no environment/Dockerfile)"
+    solve = task.solve_sh or "(no solution/solve.sh)"
+    raw = task.task_toml_raw or "(no task.toml)"
+    return (overview_md, instruction, patch, test, dockerfile, solve, raw)
+
+
+def _overview_markdown(task: HarborTask) -> str:
+    """Render the task's metadata as a clean markdown table."""
+    rows: list[tuple[str, str]] = []
+    rows.append(("task id", f"`{task.id}`"))
+    if task.name:
+        rows.append(("name", f"`{task.name}`"))
+    if task.version:
+        rows.append(("spec version", task.version))
+    if task.description:
+        rows.append(("description", task.description))
+    if task.difficulty:
+        rows.append(("difficulty", task.difficulty))
+    if task.category:
+        rows.append(("category", task.category))
+    if task.keywords:
+        rows.append(("keywords", ", ".join(f"`{k}`" for k in task.keywords)))
+    if task.agent_timeout_sec is not None:
+        rows.append(("agent timeout", f"{task.agent_timeout_sec}s"))
+    if task.verifier_timeout_sec is not None:
+        rows.append(("verifier timeout", f"{task.verifier_timeout_sec}s"))
+
+    md = ["| Field | Value |", "|---|---|"]
+    for k, v in rows:
+        md.append(f"| **{k}** | {v} |")
+
+    if task.repo2env:
+        md.append("\n### `[metadata.repo2env]` extension (Repo2RLEnv)\n")
+        md.append("| Field | Value |")
+        md.append("|---|---|")
+        for k, v in sorted(task.repo2env.items()):
+            if isinstance(v, dict):
+                # Nested pipeline-specific block — flatten one level
+                md.append(f"| **{k}** | _(nested — see below)_ |")
+                for kk, vv in sorted(v.items()):
+                    md.append(f"| &nbsp;&nbsp;`{kk}` | `{_short(vv)}` |")
+            else:
+                md.append(f"| **{k}** | `{_short(v)}` |")
+
+    return "\n".join(md)
+
+
+def _short(v) -> str:
+    """Truncate long values for the metadata table cells."""
+    if isinstance(v, list):
+        return ", ".join(str(x) for x in v)
+    s = str(v)
+    return s if len(s) < 110 else s[:107] + "…"
+
+
+# ---------------------------------------------------------------------------
+# UI
+# ---------------------------------------------------------------------------
+
+
+_INTRO_MD = """# Harbor Visualiser
+
+Browse [Harbor](https://www.harborframework.com/) task spec datasets — Hugging Face, GitHub, Harbor registry, or local."""
+
+
+_FOOTER_MD = """<sub>Built with [Gradio](https://www.gradio.app/) · Source on [GitHub](https://github.com/adithya-s-k/harbor-visualiser) · Harbor framework [docs](https://www.harborframework.com/)</sub>"""
+
+
+# A small set of popular / known-working datasets surfaced as one-click examples.
+# Each tuple is (label, uri). Order matters — most useful first.
+_EXAMPLES: list[tuple[str, str]] = [
+    ("cookbook/test (Harbor)", "harbor://cookbook/test"),
+    ("SWE-Atlas QnA (Harbor)", "harbor://scale-ai/swe-atlas-qna"),
+    ("SWE-Bench Pro (Harbor)", "harbor://cais/swebenchpro"),
+    ("Click PRs (HF / Repo2RLEnv)", "AdithyaSK/click-r2e-v082post1"),
+    ("Click PRs (GitHub demo)", "https://github.com/adithya-s-k/harbor-tasks-demo"),
+]
+
+
+# Minimal monochrome aesthetic — sharp, no rounded buttons, mono-font for code,
+# slim borders. Tuned for the Soft-replacement requested in v0.1 design.
+_CUSTOM_CSS = """
+.gradio-container { font-family: ui-sans-serif, system-ui, -apple-system, sans-serif; }
+.gradio-container * { letter-spacing: 0; }
+h1, h2, h3 { font-weight: 600; }
+button.primary { background: #111 !important; color: white !important; border: 1px solid #111 !important; }
+button.primary:hover { background: #333 !important; }
+.tabs { border: 1px solid #e5e5e5; }
+#task-list label { font-weight: 500; }
+footer { display: none !important; }
+"""
+
+
+def build_ui() -> gr.Blocks:
+    with gr.Blocks(title="Harbor Visualiser") as demo:
+        gr.Markdown(_INTRO_MD)
+
+        with gr.Row():
+            uri_input = gr.Textbox(
+                label="Dataset",
+                placeholder="owner/name  |  gh://owner/repo  |  harbor://org/name  |  https://github.com/owner/repo",
+                lines=1,
+                scale=8,
+            )
+            load_btn = gr.Button("Load", variant="primary", scale=1, min_width=80)
+
+        # Quick-access popular examples. Click → populates the input + auto-loads.
+        with gr.Row():
+            example_btns: list[gr.Button] = []
+            for label, _ in _EXAMPLES:
+                example_btns.append(gr.Button(label, size="sm", variant="secondary"))
+
+        status = gr.Markdown("Enter a dataset URI to begin.")
+
+        source_state = gr.State("")
+        root_state = gr.State("")
+
+        with gr.Row():
+            with gr.Column(scale=2, min_width=240):
+                task_list = gr.Radio(
+                    choices=[],
+                    label="Tasks",
+                    value=None,
+                    interactive=True,
+                    elem_id="task-list",
+                )
+            with gr.Column(scale=7):
+                with gr.Tabs():
+                    with gr.Tab("Overview"):
+                        overview_md = gr.Markdown()
+                    with gr.Tab("Instruction"):
+                        instruction_md = gr.Markdown()
+                    with gr.Tab("Patch"):
+                        patch_code = gr.Code(
+                            language="python",  # Gradio Prism has no "diff" but python catches +/- well
+                            label="solution/patch.diff",
+                            interactive=False,
+                        )
+                    with gr.Tab("test.sh"):
+                        test_code = gr.Code(
+                            language="shell",
+                            label="tests/test.sh",
+                            interactive=False,
+                        )
+                    with gr.Tab("Dockerfile"):
+                        dockerfile_code = gr.Code(
+                            language="dockerfile",
+                            label="environment/Dockerfile",
+                            interactive=False,
+                        )
+                    with gr.Tab("solve.sh"):
+                        solve_code = gr.Code(
+                            language="shell",
+                            label="solution/solve.sh",
+                            interactive=False,
+                        )
+                    with gr.Tab("Raw task.toml"):
+                        raw_code = gr.Code(
+                            language="yaml",  # closest Prism token to TOML
+                            label="task.toml",
+                            interactive=False,
+                        )
+
+        gr.Markdown(_FOOTER_MD)
+
+        # --- event wiring ---
+        panel_outputs = [
+            overview_md,
+            instruction_md,
+            patch_code,
+            test_code,
+            dockerfile_code,
+            solve_code,
+            raw_code,
+        ]
+
+        load_btn.click(
+            fn=load_dataset_action,
+            inputs=[uri_input],
+            outputs=[status, source_state, root_state, task_list, *panel_outputs],
+        )
+        uri_input.submit(
+            fn=load_dataset_action,
+            inputs=[uri_input],
+            outputs=[status, source_state, root_state, task_list, *panel_outputs],
+        )
+
+        task_list.change(
+            fn=select_task_action,
+            inputs=[task_list, root_state],
+            outputs=panel_outputs,
+        )
+
+        # Wire each example button → set the input AND auto-load.
+        # Capturing `uri` by default-arg avoids the classic late-binding trap.
+        for btn, (_, uri_value) in zip(example_btns, _EXAMPLES, strict=True):
+            btn.click(
+                fn=lambda u=uri_value: u,
+                outputs=uri_input,
+            ).then(
+                fn=load_dataset_action,
+                inputs=[uri_input],
+                outputs=[status, source_state, root_state, task_list, *panel_outputs],
+            )
+
+        # On page load: read ?dataset= from URL → prefill input → auto-load
+        demo.load(fn=init_from_url, inputs=None, outputs=uri_input).then(
+            fn=lambda u: load_dataset_action(u) if u else _empty_state("Enter a dataset URI to begin."),
+            inputs=[uri_input],
+            outputs=[status, source_state, root_state, task_list, *panel_outputs],
+        )
+
+    return demo
+
+
+if __name__ == "__main__":
+    # Minimal black-monochrome aesthetic. Gradio's built-in Monochrome theme
+    # is grayscale by design; combined with our css overrides above it lands
+    # at "sharp, no color, mono-ish" without rolling our own theme from scratch.
+    theme = gr.themes.Monochrome(
+        radius_size=gr.themes.sizes.radius_sm,
+        spacing_size=gr.themes.sizes.spacing_md,
+        text_size=gr.themes.sizes.text_md,
+    )
+    demo = build_ui()
+    demo.queue(default_concurrency_limit=4).launch(
+        server_name="0.0.0.0",
+        theme=theme,
+        css=_CUSTOM_CSS,
+    )

requirements.txt

@@ -0,0 +1,4 @@
+gradio>=6.0.0
+huggingface_hub>=0.27.0
+harbor>=0.6.0
+

viewer/__init__.py

@@ -0,0 +1,13 @@
+"""Harbor Visualiser — load + parse Harbor task spec datasets."""
+
+from viewer.load import DatasetSource, fetch_dataset, parse_dataset_uri
+from viewer.parse import HarborTask, list_tasks, load_task
+
+__all__ = [
+    "DatasetSource",
+    "HarborTask",
+    "fetch_dataset",
+    "list_tasks",
+    "load_task",
+    "parse_dataset_uri",
+]

viewer/load.py

@@ -0,0 +1,251 @@
+"""URI parsing + dataset fetching for the Harbor Visualiser.
+
+Accepts the same URI shapes as `repo2rlenv pull`:
+  owner/name                              → HF Hub (default)
+  owner/name@<rev>                        → HF Hub, revision pinned
+  hf://owner/name[@rev]                   → HF Hub, explicit prefix
+  gh://owner/repo[@ref]                   → GitHub, optional branch/tag/sha
+  https://github.com/owner/repo[.git]     → GitHub, full URL
+
+All datasets land in a local cache dir; subsequent loads in the same Space
+process are free.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+import shutil
+import subprocess
+import tempfile
+from dataclasses import dataclass
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+
+# Where downloads are cached during a single Space-process lifetime.
+# In Spaces, /tmp survives across requests for the same instance.
+CACHE_ROOT = Path(os.environ.get("HARBOR_VIEWER_CACHE", "/tmp/.harbor-viewer-cache"))
+
+
+@dataclass(slots=True, frozen=True)
+class DatasetSource:
+    """A parsed dataset URI ready for fetch.
+
+    `kind` is one of "hf" | "gh" | "local". `ident` is the canonical id
+    (`owner/name` for HF/GH, absolute path for local). `revision` is the
+    optional @-suffix (tag / branch / commit / Hub revision).
+    """
+
+    kind: str
+    ident: str
+    revision: str | None
+
+    @property
+    def display(self) -> str:
+        rev = f"@{self.revision}" if self.revision else ""
+        if self.kind == "hf":
+            return f"hf://{self.ident}{rev}"
+        if self.kind == "gh":
+            return f"gh://{self.ident}{rev}"
+        if self.kind == "harbor":
+            return f"harbor://{self.ident}{rev}"
+        return str(self.ident)
+
+    @property
+    def cache_key(self) -> str:
+        rev = (self.revision or "head").replace("/", "_")
+        return f"{self.kind}__{self.ident.replace('/', '__')}__{rev}"
+
+
+# ---------------------------------------------------------------------------
+# URI parsing
+# ---------------------------------------------------------------------------
+
+
+def _split_revision(s: str) -> tuple[str, str | None]:
+    """Split `name@rev` into (name, rev). Returns (name, None) if no `@`."""
+    if "@" in s:
+        name, _, rev = s.rpartition("@")
+        return name, (rev or None)
+    return s, None
+
+
+def parse_dataset_uri(uri: str) -> DatasetSource:
+    """Classify a URI string into a DatasetSource. Raises ValueError on malformed input."""
+    s = (uri or "").strip()
+    if not s:
+        raise ValueError("empty dataset URI")
+
+    # Absolute / relative path → local
+    if s.startswith("/") or s.startswith("./") or s.startswith("../"):
+        path = Path(s).expanduser().resolve()
+        if not path.is_dir():
+            raise ValueError(f"local dataset directory not found: {path}")
+        return DatasetSource(kind="local", ident=str(path), revision=None)
+
+    # GitHub full URL
+    for prefix in ("https://github.com/", "http://github.com/", "git@github.com:"):
+        if s.startswith(prefix):
+            tail = s.removeprefix(prefix).removesuffix(".git")
+            base, rev = _split_revision(tail)
+            parts = [p for p in base.split("/") if p]
+            if len(parts) < 2:
+                raise ValueError(f"GitHub URL needs owner/repo, got {uri!r}")
+            return DatasetSource(kind="gh", ident=f"{parts[0]}/{parts[1]}", revision=rev)
+
+    if s.startswith("gh://"):
+        base, rev = _split_revision(s.removeprefix("gh://"))
+        parts = [p for p in base.split("/") if p]
+        if len(parts) != 2 or not all(parts):
+            raise ValueError(f"gh:// expects owner/repo, got {uri!r}")
+        return DatasetSource(kind="gh", ident=f"{parts[0]}/{parts[1]}", revision=rev)
+
+    if s.startswith("harbor://"):
+        base, rev = _split_revision(s.removeprefix("harbor://"))
+        parts = [p for p in base.split("/") if p]
+        # Harbor accepts both bare-name and org/name
+        if len(parts) == 2 and base.count("/") == 1:
+            return DatasetSource(kind="harbor", ident=f"{parts[0]}/{parts[1]}", revision=rev)
+        if len(parts) == 1 and "/" not in base:
+            return DatasetSource(kind="harbor", ident=parts[0], revision=rev)
+        raise ValueError(
+            f"harbor:// expects 'name' or 'org/name' (optionally @tag), got {uri!r}"
+        )
+
+    if s.startswith("hf://"):
+        s = s.removeprefix("hf://")
+        # fall through to HF parsing
+
+    base, rev = _split_revision(s)
+    parts = [p for p in base.split("/") if p]
+    if len(parts) == 2 and base.count("/") == 1:
+        return DatasetSource(kind="hf", ident=f"{parts[0]}/{parts[1]}", revision=rev)
+    raise ValueError(
+        f"unrecognized dataset URI {uri!r}. "
+        f"Accepted: owner/name, hf://owner/name[@rev], gh://owner/repo[@ref], "
+        f"https://github.com/owner/repo, or an absolute local path."
+    )
+
+
+# ---------------------------------------------------------------------------
+# Fetching
+# ---------------------------------------------------------------------------
+
+
+def _fetch_hf(source: DatasetSource, force: bool) -> Path:
+    """Snapshot-download an HF Hub dataset into the cache."""
+    from huggingface_hub import snapshot_download
+
+    target = CACHE_ROOT / source.cache_key
+    if not force and target.exists() and any(target.iterdir()):
+        logger.info("hf cache hit: %s", target)
+        return target
+
+    if target.exists():
+        shutil.rmtree(target)
+    target.mkdir(parents=True, exist_ok=True)
+    # Public datasets work without a token; private ones rely on $HF_TOKEN
+    # being set in the Space's secrets.
+    token = os.environ.get("HF_TOKEN") or None
+    snapshot_download(
+        repo_id=source.ident,
+        repo_type="dataset",
+        revision=source.revision,
+        local_dir=str(target),
+        token=token,
+    )
+    return target
+
+
+def _fetch_harbor(source: DatasetSource, force: bool) -> Path:
+    """Shell out to `harbor datasets download` to fetch a Harbor-registry dataset.
+
+    Harbor handles its own registry resolution, auth, and tag pinning via
+    `<org>/<name>@<tag>`. We just orchestrate + flatten the result into a
+    standard dataset layout.
+    """
+    target = CACHE_ROOT / source.cache_key
+    if not force and target.exists() and any(target.iterdir()):
+        logger.info("harbor cache hit: %s", target)
+        return target
+
+    if not shutil.which("harbor"):
+        raise RuntimeError(
+            "`harbor` CLI not on PATH. "
+            "It's listed in `requirements.txt` — on a Hugging Face Space it "
+            "installs automatically. Locally: `pip install harbor`."
+        )
+
+    if target.exists():
+        shutil.rmtree(target)
+    target.mkdir(parents=True, exist_ok=True)
+
+    selector = source.ident + (f"@{source.revision}" if source.revision else "")
+    with tempfile.TemporaryDirectory(prefix="harbor-viewer-harbor-") as tmp:
+        args = ["harbor", "datasets", "download", selector, "-o", tmp]
+        logger.info("running: %s", " ".join(args))
+        proc = subprocess.run(args, capture_output=True, text=True, timeout=600, check=False)
+        if proc.returncode != 0:
+            shutil.rmtree(target, ignore_errors=True)
+            raise RuntimeError(
+                f"harbor download failed (exit {proc.returncode}): "
+                f"{proc.stderr.strip()[:400] or proc.stdout.strip()[:400]}"
+            )
+        # If the downloaded tree has exactly one subdirectory and no task.toml at the
+        # top, recurse one level — that's harbor's typical layout (`<tmp>/<name>/...`).
+        downloaded = Path(tmp)
+        children = [c for c in downloaded.iterdir() if c.is_dir()]
+        if len(children) == 1 and not (downloaded / "task.toml").exists():
+            downloaded = children[0]
+        # Move everything under the cache target
+        for child in downloaded.iterdir():
+            shutil.move(str(child), str(target / child.name))
+
+    return target
+
+
+def _fetch_github(source: DatasetSource, force: bool) -> Path:
+    """git clone --depth 1 a GitHub repo into the cache."""
+    target = CACHE_ROOT / source.cache_key
+    if not force and target.exists() and any(target.iterdir()):
+        logger.info("gh cache hit: %s", target)
+        return target
+
+    if target.exists():
+        shutil.rmtree(target)
+    target.parent.mkdir(parents=True, exist_ok=True)
+
+    if not shutil.which("git"):
+        raise RuntimeError(
+            "`git` not found on PATH. Install git (or rely on the HF Space sandbox which ships it)."
+        )
+
+    with tempfile.TemporaryDirectory(prefix="harbor-viewer-clone-") as tmp:
+        tmp_clone = Path(tmp) / "clone"
+        args = ["git", "clone", "--depth", "1"]
+        if source.revision:
+            args += ["--branch", source.revision]
+        args += [f"https://github.com/{source.ident}.git", str(tmp_clone)]
+        logger.info("running: git clone --depth 1 [...] %s", source.ident)
+        proc = subprocess.run(args, capture_output=True, text=True, timeout=300, check=False)
+        if proc.returncode != 0:
+            raise RuntimeError(
+                f"git clone failed (exit {proc.returncode}): {proc.stderr.strip()[:400]}"
+            )
+        shutil.move(str(tmp_clone), str(target))
+    return target
+
+
+def fetch_dataset(source: DatasetSource, *, force: bool = False) -> Path:
+    """Materialize a dataset into the local cache. Returns the on-disk root."""
+    if source.kind == "hf":
+        return _fetch_hf(source, force=force)
+    if source.kind == "gh":
+        return _fetch_github(source, force=force)
+    if source.kind == "harbor":
+        return _fetch_harbor(source, force=force)
+    if source.kind == "local":
+        return Path(source.ident)
+    raise ValueError(f"unknown source kind: {source.kind!r}")

viewer/parse.py

@@ -0,0 +1,159 @@
+"""Walk a Harbor dataset directory + load each task's spec files.
+
+A "Harbor task" is a directory containing `task.toml` at its root, plus
+any of: `instruction.md`, `solution/patch.diff`, `solution/solve.sh`,
+`tests/test.sh`, `environment/Dockerfile`. We tolerate missing files —
+not every task ships every artifact.
+
+Two dataset layouts are accepted:
+
+    flat:           <root>/<task-id>/task.toml
+    nested:         <root>/tasks/<task-id>/task.toml
+
+We discover both transparently.
+"""
+
+from __future__ import annotations
+
+import logging
+import tomllib
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass(slots=True)
+class HarborTask:
+    """The full parsed view of a single Harbor task directory."""
+
+    # Identity
+    id: str  # the directory name, e.g. "pallets__click-3373"
+    root: Path  # absolute path to the task dir
+
+    # task.toml top-level fields
+    name: str | None = None
+    org: str | None = None
+    version: str | None = None
+
+    # [task]
+    description: str | None = None
+    instruction_inline: str | None = None  # if task.toml has `instruction =`
+
+    # [metadata]
+    difficulty: str | None = None
+    category: str | None = None
+    keywords: list[str] = field(default_factory=list)
+
+    # [agent] / [verifier] timeouts (if set)
+    agent_timeout_sec: float | None = None
+    verifier_timeout_sec: float | None = None
+
+    # Repo2RLEnv extension — [metadata.repo2env] block, opaque dict
+    repo2env: dict[str, Any] | None = None
+
+    # File contents (None = file not present)
+    instruction_md: str | None = None
+    oracle_patch: str | None = None
+    solve_sh: str | None = None
+    test_sh: str | None = None
+    dockerfile: str | None = None
+    task_toml_raw: str = ""
+
+
+def _read_text(path: Path) -> str | None:
+    """Read a text file, return None if it doesn't exist."""
+    try:
+        return path.read_text(encoding="utf-8")
+    except FileNotFoundError:
+        return None
+    except Exception as exc:
+        logger.warning("could not read %s: %s", path, exc)
+        return None
+
+
+def _discover_task_roots(dataset_root: Path) -> list[Path]:
+    """Find every directory under `dataset_root` that contains a `task.toml`.
+
+    Handles both flat (`<root>/<id>/task.toml`) and nested
+    (`<root>/tasks/<id>/task.toml`) Harbor layouts. Prefers nested when
+    `<root>/tasks/` exists (that's what `repo2rlenv push` stages on HF).
+    Also handles the case where the dataset is itself a single task root.
+    """
+    if (dataset_root / "task.toml").exists():
+        return [dataset_root]
+
+    tasks_dir = dataset_root / "tasks"
+    if tasks_dir.is_dir():
+        return sorted(
+            [p for p in tasks_dir.iterdir() if p.is_dir() and (p / "task.toml").exists()]
+        )
+
+    # Flat layout — every immediate subdir that has task.toml is a task
+    return sorted(
+        [
+            p
+            for p in dataset_root.iterdir()
+            if p.is_dir() and not p.name.startswith(".") and (p / "task.toml").exists()
+        ]
+    )
+
+
+def list_tasks(dataset_root: Path) -> list[str]:
+    """Return the task-id (directory name) of every task under `dataset_root`."""
+    return [p.name for p in _discover_task_roots(dataset_root)]
+
+
+def _resolve_task_dir(dataset_root: Path, task_id: str) -> Path:
+    """Find the on-disk directory for a given task id (handles flat + nested)."""
+    flat = dataset_root / task_id
+    if (flat / "task.toml").exists():
+        return flat
+    nested = dataset_root / "tasks" / task_id
+    if (nested / "task.toml").exists():
+        return nested
+    if (dataset_root / "task.toml").exists() and dataset_root.name == task_id:
+        return dataset_root
+    raise FileNotFoundError(f"no task {task_id!r} under {dataset_root}")
+
+
+def load_task(dataset_root: Path, task_id: str) -> HarborTask:
+    """Load every spec file for a single Harbor task. Tolerant of missing pieces."""
+    task_dir = _resolve_task_dir(dataset_root, task_id)
+    toml_path = task_dir / "task.toml"
+    raw = toml_path.read_text(encoding="utf-8")
+    data = tomllib.loads(raw)
+
+    task_block = data.get("task") or {}
+    metadata_block = data.get("metadata") or {}
+    agent_block = data.get("agent") or {}
+    verifier_block = data.get("verifier") or {}
+
+    # [metadata.repo2env] is the Repo2RLEnv extension — surfaced as a separate
+    # opaque dict so the UI can render it specially if present.
+    repo2env = metadata_block.get("repo2env")
+    if repo2env is not None and not isinstance(repo2env, dict):
+        repo2env = None
+
+    return HarborTask(
+        id=task_id,
+        root=task_dir,
+        name=task_block.get("name"),
+        org=task_block.get("org"),
+        version=data.get("version"),
+        description=task_block.get("description"),
+        instruction_inline=task_block.get("instruction"),
+        difficulty=metadata_block.get("difficulty"),
+        category=metadata_block.get("category"),
+        keywords=list(metadata_block.get("keywords") or []),
+        agent_timeout_sec=agent_block.get("timeout_sec"),
+        verifier_timeout_sec=verifier_block.get("timeout_sec"),
+        repo2env=repo2env,
+        instruction_md=_read_text(task_dir / "instruction.md"),
+        oracle_patch=_read_text(task_dir / "solution" / "patch.diff"),
+        solve_sh=_read_text(task_dir / "solution" / "solve.sh"),
+        test_sh=_read_text(task_dir / "tests" / "test.sh"),
+        dockerfile=_read_text(task_dir / "environment" / "Dockerfile"),
+        task_toml_raw=raw,
+    )