AnyCrawler Read

AnyCrawler Read

Use this skill when an agent needs webpage content with low context overhead. Prefer the bundled CLI in scripts/anycrawler_crawl_api.py. When the skill is installed under ~/.codex/skills/anycrawler-read, the CLI checks the canonical GitHub release tags once per agent session and auto-updates before continuing if the install is behind. Developer checkouts do not self-mutate.

Preconditions

1. page --method fetch uses the free fetch endpoint and does not require an API key.
2. page --method render and screenshot require ANYCRAWLER_API_KEY.
3. ANYCRAWLER_BASE_URL is optional; default is https://api.anycrawler.com.
4. Use documented snake_case fields only.

Choose the endpoint

• Use page for reading, summarizing, or extracting webpage content.
• Use screenshot only when the task explicitly needs a PNG or snapshot_url.

Choose the method

• Start with page --method fetch; it calls GET /free/v1/crawl?url=... without authentication.
• Switch to page --method render when fetched output is incomplete or the page depends on client-side rendering.
• If async content still has not settled, add --browser-wait-until networkidle2 with render.

Common commands

python scripts/anycrawler_crawl_api.py page \
  --url https://example.com \
  --write-markdown out.md

python scripts/anycrawler_crawl_api.py screenshot \
  --url https://example.com \
  --download-snapshot snapshot.png

Request rules

• page --method fetch sends only url to the free endpoint.
• page --method render supports url, method, include_metadata, include_links, include_media, markdown_variant, and browser_wait_until.
• accept_cache applies only when method=render; do not send or recommend it for method=fetch.
• browser_wait_until applies only when method=render.
• include_metadata, include_links, and include_media only affect authenticated render responses when explicitly enabled.
• markdown_variant=readability still returns content in results.markdown.
• Do not rely on undocumented passthrough fields.

Response handling

Inspect both data and meta.

• Success path:
  • Check data.ok.
  • Read markdown from data.results.markdown.
  • Read optional data.results.metadata, data.results.links, and data.results.media only if those sections were requested.
  • For screenshots, use data.results.snapshot_url or --download-snapshot.
• Failure path:
  • Record meta.requestId.
  • The CLI may mirror extra meta fields, but agents usually only need requestId for troubleshooting.
  • Check data.error and data.retryable.

Retry rules

• Do not blindly retry 400, 401, 402, or most 403 responses.
• Treat 403 from paid-plan-only fields as a request-shape issue; remove those fields before retrying.
• 409, 429, 502, and 504 are the main backoff-and-retry cases.
• For 429, treat it as quota, rate-limit, or concurrency pressure first; back off before retrying.

References

• Read references/public-api.md for the minimal public contract.
• Read references/maintainer.md only for release, billing, and full gateway details.