Skip to main content
POST
Adds a domain crawl source and queues discovery/indexing. Returns 201 Created.

Authentication

X-API-Key
string
required
Project API key for this organization and project. Format: bot_live_…

Path parameters

organization_id
string
required
Organization ID. Must match the API key.
project_id
string
required
Project ID. Must match the API key.

Body

domain
string
required
Domain or site root to crawl (for example https://example.com).
crawling_strategy
string
default:"deep_crawl"
sitemap or deep_crawl.
max_pages
integer
default:"50"
Maximum pages to crawl. Range: 11000.
max_depth
integer
default:"3"
Maximum crawl depth. Range: 110.
include_patterns
array
Whitelist regex/glob patterns; only matching URLs are crawled.
exclude_patterns
array
Patterns to exclude from the crawl.
frequency_days
integer
default:"30"
Automatic recrawl interval in days (1365).
crawler
string
default:"crawl4ai"
Crawler engine: crawl4ai or apify.

Response

Success message confirming the domain crawl was queued. data is typically null.

Errors

401
Unauthorized
Missing, invalid, or revoked API key.
403
Forbidden
API key is not authorized, or link quota exceeded.