Website Crawling

Use Data Sources -> Website to ingest website content.

Configuration

Set the website URL and optional page limit.

Default max pages: 100
Setup discovery scans: 300
Maximum accepted in UI: 5000
Non-root URLs scope crawling to that path prefix. For example, https://support.example.com/en/ crawls only support.example.com/en and nested pages under /en.

After saving, a crawl/index process runs and status updates are visible in the website dashboard.

Crawl Status Stages

Typical lifecycle:

queued
crawling
indexing
complete
error

The UI surfaces pages crawled, indexed pages, and chunk totals while the process runs.

If a scan reaches its configured page limit, the status shows that the limit was reached. Full site scans beyond the configured limit require contacting support.

Recommended Scope Strategy

Start with full website for initial launch.
Use nested paths for domain-specific agents.
Exclude low-value pages by disabling pages during review.

Quality Tips

Prioritize pages with definitive product, policy, and contact information.
Re-crawl after major site updates.
Use FAQ snippets for policy answers that must be exact.