unbrowse

Unbrowse — Website-to-API Reverse Engineering

Overview

Unbrowse is a local service backed by a shared skill marketplace. When you ask it to do something, it first searches the marketplace for an existing skill discovered by any agent. If none exists, it captures the site, reverse-engineers the API, publishes the skill to the marketplace, and executes it. Future requests from any agent reuse the learned skill instantly.

The local server runs on http://localhost:6969 (or $UNBROWSE_URL if configured) and proxies marketplace operations to beta-api.unbrowse.ai. On first startup it auto-registers as an agent and caches the API key in ~/.unbrowse/config.json.

How Intent Resolution Works

When you call POST /v1/intent/resolve, the orchestrator follows this priority chain:

1. Marketplace search -- Semantic vector search for existing skills matching your intent. Candidates are ranked by composite score: 40% embedding similarity + 30% reliability + 15% freshness + 15% verification status. If a skill scores above the confidence threshold, it executes immediately.
2. Live capture -- If no marketplace skill matches, a headless browser navigates to the URL, records all network traffic, reverse-engineers API endpoints, and publishes a new skill to the marketplace.
3. DOM fallback -- If no API endpoints are found (static/SSR sites), structured data is extracted from the rendered HTML.

Skills published by live capture become available to all agents on the network.

Quick Start

Set the base URL:

UNBROWSE=${UNBROWSE_URL:-http://localhost:6969}

Server Startup (Auto-handled)

Before making any API calls, ensure the local server is running. Check health first:

curl -sf "$UNBROWSE/health" || echo "NOT_RUNNING"

If the server is not running, start it. On first run it needs to register and accept the Terms of Service.

First-time setup (requires user consent):

1. Show the user the ToS summary and ask for their explicit consent:

   Unbrowse needs you to accept its Terms of Service before it can run:

   • API structures you discover may be shared in the collective registry
   • You will not use Unbrowse to attack, overload, or abuse any target site
   • The service is provided "as is" without warranty
   • Unreel AI Pte Ltd (Singapore) governs these terms Full terms: https://unbrowse.ai/terms

   Do you accept?

2. Only after the user agrees, start the server with their consent acknowledged:

cd ~/.agents/skills/unbrowse && UNBROWSE_NON_INTERACTIVE=1 UNBROWSE_TOS_ACCEPTED=1 nohup bun src/index.ts > /tmp/unbrowse.log 2>&1 &

3. Wait for it to be ready:

for i in $(seq 1 10); do curl -sf "$UNBROWSE/health" && break || sleep 1; done

If the user declines, do not start the server. Unbrowse cannot operate without ToS acceptance.

Subsequent starts (already registered — ~/.unbrowse/config.json exists):

cd ~/.agents/skills/unbrowse && UNBROWSE_NON_INTERACTIVE=1 UNBROWSE_TOS_ACCEPTED=1 nohup bun src/index.ts > /tmp/unbrowse.log 2>&1 &

Agent Registration (Automatic)

The local server auto-registers on first startup and caches credentials in ~/.unbrowse/config.json. No manual API key setup is needed — it handles registration on boot once ToS is accepted.

Core Workflow

1. Natural Language Intent Resolution (Recommended)

The simplest way -- describe what you want and unbrowse figures out the rest:

curl -s -X POST "$UNBROWSE/v1/intent/resolve" \
  -H "Content-Type: application/json" \
  -d '{"intent": "get trending searches on Google", "params": {"url": "https://google.com"}, "context": {"url": "https://google.com"}}'

This will: search the marketplace for a matching skill, or capture the site, extract API endpoints, learn a skill, publish it, and execute it -- all in one call.

2. Manual Capture -> Execute Flow

Step 1: Capture a website

curl -s -X POST "$UNBROWSE/v1/intent/resolve" \
  -H "Content-Type: application/json" \
  -d '{"intent": "capture APIs from this site", "params": {"url": "https://example.com"}, "context": {"url": "https://example.com"}}'

Step 2: List learned skills

curl -s "$UNBROWSE/v1/skills" | jq .

Step 3: Execute a specific skill

curl -s -X POST "$UNBROWSE/v1/skills/{skill_id}/execute" \
  -H "Content-Type: application/json" \
  -d '{"params": {}}'

Step 4: Inspect endpoint schema

curl -s "$UNBROWSE/v1/skills/{skill_id}/endpoints/{endpoint_id}/schema" | jq .

Authentication for Gated Sites

If a site requires login:

Interactive Login (opens a browser window)

curl -s -X POST "$UNBROWSE/v1/auth/login" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/login"}'

The user completes login in the browser. Cookies are stored in the vault and automatically used for subsequent captures and executions on that domain.

Yolo Login (use existing Chrome sessions)

If the user is already logged into a site in their main Chrome browser, yolo mode opens Chrome with their real profile -- no need to re-login.

Important: Always ask the user before using yolo mode. Say: "I'll open your main Chrome browser with all your existing sessions. You'll need to close Chrome first. OK to proceed?"

curl -s -X POST "$UNBROWSE/v1/auth/login" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "yolo": true}'

If the response contains "Chrome is running" error, tell the user to close Chrome and retry.

After Login, Re-capture

curl -s -X POST "$UNBROWSE/v1/intent/resolve" \
  -H "Content-Type: application/json" \
  -d '{"intent": "get my dashboard data", "params": {"url": "https://example.com/dashboard"}, "context": {"url": "https://example.com"}}'

Stored auth cookies are automatically loaded from the vault.

Mutation Safety

For non-GET endpoints (POST, PUT, DELETE), unbrowse requires explicit confirmation:

Dry Run (preview what would execute)

curl -s -X POST "$UNBROWSE/v1/skills/{skill_id}/execute" \
  -H "Content-Type: application/json" \
  -d '{"params": {}, "dry_run": true}'

Confirm Unsafe Execution

curl -s -X POST "$UNBROWSE/v1/skills/{skill_id}/execute" \
  -H "Content-Type: application/json" \
  -d '{"params": {}, "confirm_unsafe": true}'

Always use dry_run first for mutations. Ask the user before passing confirm_unsafe.

Field Projection

Request only specific fields from the response:

curl -s -X POST "$UNBROWSE/v1/skills/{skill_id}/execute" \
  -H "Content-Type: application/json" \
  -d '{"params": {}, "projection": {"include": ["title", "url", "score"]}}'

Feedback

Report whether a skill execution was useful:

curl -s -X POST "$UNBROWSE/v1/feedback" \
  -H "Content-Type: application/json" \
  -d '{"target_type": "skill", "target_id": "{skill_id}", "endpoint_id": "{endpoint_id}", "outcome": "success", "rating": 5}'

Ratings (1-5) affect the skill's reliability score and marketplace ranking. Skills with consistently low ratings or consecutive execution failures are automatically deprecated from the marketplace.

Reporting Issues

If a skill is broken or returns wrong data, report it:

curl -s -X POST "$UNBROWSE/v1/skills/{skill_id}/issues" \
  -H "Content-Type: application/json" \
  -d '{"category": "broken", "description": "Endpoint returns 403", "endpoint_id": "{endpoint_id}"}'

Categories: broken, wrong_data, needs_auth, rate_limited, stale_schema, missing_endpoint, other.

Search the Marketplace

Global search — find skills by intent across all domains

curl -s -X POST "$UNBROWSE/v1/search" \
  -H "Content-Type: application/json" \
  -d '{"intent": "get product prices", "k": 5}'

Domain-scoped search — find skills for a specific site

curl -s -X POST "$UNBROWSE/v1/search/domain" \
  -H "Content-Type: application/json" \
  -d '{"intent": "get trending items", "domain": "amazon.com", "k": 5}'

Response:

{
  "results": [
    {"id": 1, "score": 0.92, "metadata": {"skill_id": "...", "domain": "amazon.com", "name": "..."}}
  ]
}

Use the returned skill_id to execute the skill directly via /v1/skills/{skill_id}/execute.

Platform Stats

Get aggregate marketplace statistics:

curl -s "$UNBROWSE/v1/stats/summary" | jq .

Response:

{"skills": 142, "endpoints": 580, "domains": 67, "executions": 3200, "agents": 45}

Agent Profiles

Get your own profile (authenticated)

curl -s -H "Authorization: Bearer $UNBROWSE_API_KEY" "$UNBROWSE/v1/agents/me" | jq .

Get any agent's public profile

curl -s "$UNBROWSE/v1/agents/{agent_id}" | jq .

List recent agents

curl -s "$UNBROWSE/v1/agents?limit=20" | jq .

Response:

{
  "agent_id": "abc123",
  "name": "my-agent",
  "created_at": "2025-01-15T10:00:00Z",
  "skills_discovered": ["skill_abc", "skill_def"],
  "total_executions": 47,
  "total_feedback_given": 12
}

Skill Verification

Trigger a health check on a skill's endpoints:

curl -s -X POST "$UNBROWSE/v1/skills/{skill_id}/verify" | jq .

Endpoint Selection

When intent/resolve returns, the response includes an available_endpoints array listing all discovered endpoints. The auto-selected endpoint may not always be the best one for your intent.

If the result looks wrong (e.g. you got a config blob, tracking data, or the wrong page), look at available_endpoints and re-execute with the correct one:

curl -s -X POST "$UNBROWSE/v1/skills/{skill_id}/execute" \
  -H "Content-Type: application/json" \
  -d '{"params": {"endpoint_id": "{correct_endpoint_id}"}}'

How to pick the right endpoint:

• Prefer endpoints whose URL path matches your intent (e.g. /quotes for quotes, /api/products for products)
• Endpoints with `dom_extract

---

Content truncated.