Crawled-pages list / crawler logs
The complete per-URL list of URLs AI bots fetched — the public twin of GET /api/v1/me/brands//agents/pages. EVERY crawled URL is included: static assets (/_next/..., images, CSS) and crawler-directive files (/robots.txt, /llms.txt) are reported alongside content pages, because what a crawler spends its budget on is the point of the report. Paginated, searchable, and filterable by status (ok, error, or one exact code such as 404), bot, vendor, kind, and folder. Crawler filters are served from the (path x bot) rollup, which starts at the raw-hit retention horizon and grows from there — see retention_limited / coverage_from.
Authorizations
API keys generated via POST /api/v1/me/api-keys. Format:
Authorization: Bearer fxa_<32-char-secret>. Only accepted on
/api/public/v1/*. Cookie auth is rejected on that surface.
Path Parameters
URL-safe brand identifier (lowercased, hyphenated).
^[a-z0-9-]+$Query Parameters
Time window for time-series + aggregations.
7d, 30d, 90d Exact bot name (e.g. GPTBot). Part of the page-wide crawler filter: every aggregate on the Agents dashboard honours it, so the KPIs, the trend, the breakdowns and the pages list always describe the same traffic. Combine with vendor/kind to narrow further (they AND).
Exact vendor / platform (e.g. OpenAI) — every bot that company operates.
Exact agent type, e.g. training_crawler for bots that collect data for model training vs browsing_agent for live answer-time fetches.
training_crawler, search_crawler, browsing_agent, agentic_browser, mcp_client Top-level folder to restrict to, e.g. blogs for /blogs/*, or / for pages at the site root.
Column to order by. Defaults to hits. Unrecognised values fall back to the default rather than erroring.
hits, path, errors, uniques Sort direction. Only honoured alongside sort. Defaults per column: descending for the numeric columns, ascending for path.
asc, desc x <= 500Response
A page of crawled content pages + the total distinct count.