Skip to main content
GET
Crawled-pages list / crawler logs

Authorizations

Authorization
string
header
required

API keys generated via POST /api/v1/me/api-keys. Format: Authorization: Bearer fxa_<32-char-secret>. Only accepted on /api/public/v1/*. Cookie auth is rejected on that surface.

Path Parameters

slug
string
required

URL-safe brand identifier (lowercased, hyphenated).

Pattern: ^[a-z0-9-]+$

Query Parameters

range
enum<string>
default:30d

Time window for time-series + aggregations.

Available options:
7d,
30d,
90d
status
string
bot
string

Exact bot name (e.g. GPTBot). Part of the page-wide crawler filter: every aggregate on the Agents dashboard honours it, so the KPIs, the trend, the breakdowns and the pages list always describe the same traffic. Combine with vendor/kind to narrow further (they AND).

vendor
string

Exact vendor / platform (e.g. OpenAI) — every bot that company operates.

kind
enum<string>

Exact agent type, e.g. training_crawler for bots that collect data for model training vs browsing_agent for live answer-time fetches.

Available options:
training_crawler,
search_crawler,
browsing_agent,
agentic_browser,
mcp_client
folder
string

Top-level folder to restrict to, e.g. blogs for /blogs/*, or / for pages at the site root.

sort
enum<string>

Column to order by. Defaults to hits. Unrecognised values fall back to the default rather than erroring.

Available options:
hits,
path,
errors,
uniques
dir
enum<string>

Sort direction. Only honoured alongside sort. Defaults per column: descending for the numeric columns, ascending for path.

Available options:
asc,
desc
limit
integer
default:10
Required range: x <= 500
offset
integer
default:0

Response

200

A page of crawled content pages + the total distinct count.