Crawling websites

POST /v1/ingest/crawl starts a crawl of one or more public websites. The crawler follows links from your seed URLs within the hosts you allow, converts each page to text, and indexes it as a document in the namespace you choose, exactly as an uploaded file would be. Use it for the content you publish anyway: court locations and hours, fee schedules, jury-duty FAQs, self-help pages.

Requires the ingest scope. Request bodies are JSON and subject to the 5 MiB body cap (Conventions).

Start a crawl

curl -s "https://api.chatbots.ecourtdate.com/v1/ingest/crawl" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "public-site",
    "seed_urls": [
      "https://courts.example.gov/self-help",
      "https://courts.example.gov/jury-duty"
    ],
    "allowed_domains": ["courts.example.gov"],
    "max_pages": 200,
    "max_depth": 3,
    "rate_limit_rps": 2.0,
    "respect_robots_txt": true
  }'

Response (200):

{
  "crawl_job_id": "4631b07b-8c2d-4f1e-9a6b-3d5e7f9a1b2c",
  "status": "processing"
}

The response returns as soon as the job is queued; no page has been fetched yet. Poll GET /v1/ingest/crawl/{crawlJobId} to follow progress.

Request

Field	Type	Description
`namespace`	string	Required. The knowledge-base namespace that receives the crawled documents. Must match `^[a-z0-9][a-z0-9_-]{0,63}$` (lowercase letters, digits, `_` and `-`, 1 to 64 characters); an invalid value is a `400` with `param: namespace` and the message `String should match pattern '^[a-z0-9][a-z0-9_-]{0,63}$'`. Pick the namespace of the bot that should answer from this site (Namespaces).
`seed_urls`	string[]	Required. 1 to 100 `http` or `https` URLs to start from. An empty list is a `400` with `param: seed_urls` (`List should have at least 1 item after validation, not 0`); more than 100 is a `400` on the same `param`. Every seed must point at a public host (Seed URL rules).
`allowed_domains`	string[]	Hosts the crawler may fetch from. Default `[]`, which derives the allowlist from the seed URLs' hosts. See Domain allowlist.
`max_pages`	integer	Stop after this many pages have been fetched successfully. 1 to 500, default `50`.
`max_depth`	integer	Maximum number of link hops from a seed (seeds are depth 0). 1 to 10, default `3`.
`rate_limit_rps`	number	Maximum fetch rate against the site, in requests per second. 0.1 to 10, default `2.0`.
`respect_robots_txt`	boolean	Honor the site's `robots.txt`. Default `true`. See robots.txt.

Values must be JSON-typed: booleans, integers, and numbers are not parsed from strings ("max_pages": "10" or "respect_robots_txt": "yes" is a 400). Unknown fields are ignored. Call the path without a trailing slash; a trailing slash redirects with 307, and clients that drop the body on a redirect then create no job.

Response

Field	Type	Description
`crawl_job_id`	string	UUID of the crawl job. Poll it at `GET /v1/ingest/crawl/{crawlJobId}`; it is also the `crawl_job_id` of the `crawl.completed` webhook.
`status`	string	Always `processing` on this response.

Seed URL rules

Every seed URL is checked when the request is submitted. A seed is rejected when:

its scheme is not http or https (the scheme is compared case-insensitively);
it embeds credentials (https://user:pass@host/);
it has no host;
its host resolves to a private, loopback, link-local, multicast, reserved, or unspecified address (127.0.0.1, 10.0.0.5, [::1], 0.0.0.0, 169.254.169.254, or any hostname whose DNS records include such an address);
its host cannot be resolved, or resolves to no address at all.

DNS is resolved at submission time, and every address the host resolves to must be public. Public IP literals (http://93.184.216.34/) are accepted without a DNS lookup.

If any seed fails, the whole request is rejected with 400 and no job is created. param names the first failing seed:

{
  "error": {
    "message": "Seed URL 'http://127.0.0.1/admin' is not allowed: Host '127.0.0.1' resolves to a disallowed address (127.0.0.1).",
    "type": "invalid_request_error",
    "param": "seed_urls[1]",
    "code": null
  }
}

The reason is one of:

Reason	Cause
`Scheme 'ftp' is not allowed (http/https only).`	Unsupported scheme.
`URLs with embedded credentials are not allowed.`	Userinfo in the URL.
`URL has no host.`	Malformed URL.
`Host 'intranet.example.gov' resolves to a disallowed address (10.1.2.3).`	A private or otherwise non-public address.
`Could not resolve host 'typo.example.gov'.`	DNS lookup failed.
`Host 'example.gov' did not resolve to any address.`	DNS returned no records.

The same address rules apply while the crawl runs: a discovered link or a redirect that points at a non-public address is discarded silently.

Domain allowlist

allowed_domains bounds the crawl. A URL is in scope only when its host (including the port when it is not the scheme default) is an exact, case-sensitive match for an entry in the list:

courts.example.gov does not match www.courts.example.gov, forms.courts.example.gov, courts.example.gov:8443, or Courts.example.gov. List every host variant you want crawled.
A scheme's default port is treated as no port: https://courts.example.gov:443/ and http://courts.example.gov:80/ both match courts.example.gov, in the list and in the allowlist derived from the seeds.
With the default [], the allowlist is derived from the hosts of the seed URLs, so a single-host crawl needs no allowed_domains at all.
With an explicit list, a seed whose host is not in the list is skipped silently: it is not fetched, produces no error, and a crawl whose seeds are all out of scope completes with 0 pages.
Discovered links are followed only when their host is in the list and they pass the address rules.
A redirect to a host outside the list (or to a non-public address) is discarded. A redirect that stays in scope is followed, and the document records the final URL.

How the crawl proceeds

The crawl is breadth-first from the seeds: all seeds (depth 0), then every in-scope link found on them (depth 1), and so on up to max_depth.
It stops once max_pages pages have been fetched successfully. Pages that the site answers with an HTTP error status (4xx or 5xx) are not indexed, even when the error page carries HTML: they are recorded in errors[] with the page URL and the error HTTP <status> (for example HTTP 404), and they do not count toward max_pages. Pages the crawler cannot fetch at all (a network error, a robots.txt denial, an empty response) are skipped silently and appear nowhere.
After each successful fetch the crawler waits 1 / rate_limit_rps seconds, so rate_limit_rps: 2.0 means at most two fetches per second against the site, and 0.5 means one every two seconds.
URLs are normalized by stripping a trailing slash (https://courts.example.gov/ is recorded as https://courts.example.gov), and a URL reached twice is fetched once.
A page that yields no readable text (an empty page, a page made only of navigation or images) counts toward pages_crawled but not pages_indexed, and creates no document.
A crawl has a 10-minute budget, fetching and indexing included. A crawl that exceeds it (or is cancelled while running) ends failed with the error Crawl timed out.; pages indexed before that moment stay ready (so pages_indexed can be greater than 0 on the failed job), and any page still being indexed is marked failed with the same error. Narrow max_pages, max_depth, or allowed_domains, or raise rate_limit_rps where the site permits, and start again.

The bounds allow requests that cannot finish inside the budget: 500 pages at rate_limit_rps: 0.1 need over 80 minutes of pacing alone. A crawl of 200 pages at the default rate_limit_rps takes at least 100 seconds of fetching before any indexing; plan on polling, or use the crawl.completed webhook.

robots.txt

With respect_robots_txt: true (the default) the crawler reads the site's robots.txt and does not fetch paths it disallows. Set it to false only for sites you operate and have the right to crawl in full; the crawler still enforces allowed_domains, the address rules, and rate_limit_rps either way.

Check the crawl status

curl -s "https://api.chatbots.ecourtdate.com/v1/ingest/crawl/4631b07b-8c2d-4f1e-9a6b-3d5e7f9a1b2c" \
  -H "Authorization: Bearer $API_KEY"

{
  "crawl_job_id": "4631b07b-8c2d-4f1e-9a6b-3d5e7f9a1b2c",
  "status": "completed_with_errors",
  "pages_crawled": 148,
  "pages_indexed": 146,
  "errors": [
    {
      "url": "https://courts.example.gov/forms/fee-waiver",
      "error": "The page could not be retrieved or contained no readable text."
    }
  ]
}

Field	Type	Description
`crawl_job_id`	string	The job's UUID.
`status`	string	`processing`, `completed`, `completed_with_errors`, or `failed`. See Job status.
`pages_crawled`	integer	Pages fetched successfully so far, including pages that produced no text.
`pages_indexed`	integer	Pages that became `ready` documents so far.
`errors`	object[]	One entry per page that could not be indexed: `url` (the page) and `error` (a sanitized reason); or a single entry with `url: crawl` when the crawl itself failed.

All three are updated as each page is processed, so you can watch a running crawl advance. Status semantics:

Status	Meaning
`processing`	Queued or running. Counts are live and may still grow.
`completed`	Finished with no page errors (pages with no readable text are not errors).
`completed_with_errors`	Finished; at least one page was indexed and at least one failed. Check `errors[]`.
`failed`	Nothing was indexed, or the crawl itself failed: an internal error, a timeout, or a cancellation. On a timeout or cancellation `pages_indexed` can still be greater than 0; those documents are `ready`.

When the crawl as a whole fails (rather than an individual page), errors[] holds a single entry whose url is the sentinel crawl:

{
  "crawl_job_id": "4631b07b-8c2d-4f1e-9a6b-3d5e7f9a1b2c",
  "status": "failed",
  "pages_crawled": 0,
  "pages_indexed": 0,
  "errors": [
    {
      "url": "crawl",
      "error": "Processing failed due to an internal error. Please retry; contact support if the problem persists."
    }
  ]
}

Per-page error strings are one of:

Error	Meaning
`HTTP <status>` (for example `HTTP 404`)	The site answered with an HTTP error status; the page was not indexed.
`The page could not be retrieved or contained no readable text.`	The page was fetched but could not be parsed into text.
`Processing failed due to an internal error. Please retry; contact support if the problem persists.`	Indexing the page failed for another reason.
`Crawl timed out.`	The crawl exceeded its 10-minute budget or was cancelled (sentinel `url: crawl`; any page still being indexed at that moment ends as a `failed` document with this error).

A page listed in errors[] also has a document in status failed carrying the same error, so GET /v1/documents?status=failed&source_type=crawl shows the same picture; the exception is a page the site answered with an HTTP error status, which was never indexed and has no document. A page with no readable text is neither an error nor a document: it counts toward pages_crawled only. Crawl jobs are not retried automatically; fix the cause and start a new crawl.

A crawl job belongs to the account whose key created it. An unknown id, or a job created under another account, returns 404 not_found. There is no endpoint to list, cancel, or delete crawl jobs.

Crawled documents

Each indexed page becomes a document in the job's namespace:

{
  "id": "c0b1f6a4-7d2e-4f3a-8b9c-1d2e3f4a5b6c",
  "object": "document",
  "namespace": "public-site",
  "filename": "https://courts.example.gov/jury-duty/reporting",
  "source_type": "crawl",
  "source_url": "https://courts.example.gov/jury-duty/reporting",
  "status": "ready",
  "chunk_count": 4,
  "error": null,
  "created_at": "2026-08-21T15:12:40.118000Z",
  "updated_at": "2026-08-21T15:12:41.507000Z"
}

For crawled documents filename and source_url are both the final page URL (after any in-scope redirect, trailing slash stripped). Chat completions cite them through source_filename in the citation object, so the page URL is what your users see.

Re-crawling

Crawls are not deduplicated against earlier crawls. Running the same crawl again creates a second document for every page, and both copies are retrieved and cited. To refresh a site:

Start the new crawl and wait for it to complete.
List the old documents with GET /v1/documents?namespace=public-site&source_type=crawl and delete those whose created_at predates the new job.

Or crawl into a fresh namespace and ask eCourtDate to point the bot at it once the crawl completes, which avoids any window with stale or duplicated answers.

Errors

Status	Code	When
`400`	`null`	A field is missing, out of range, wrongly typed, or does not match its pattern (`param` names it, for example `namespace`, `max_pages`, or `seed_urls`), or a seed URL was rejected (`param: seed_urls[i]`, see Seed URL rules).
`401`	`invalid_api_key`	Missing or invalid key.
`403`	`insufficient_scope`	The key lacks the `ingest` scope.
`404`	`not_found`	`GET /v1/ingest/crawl/{crawlJobId}` for an unknown id or another account's job.
`413`	`request_too_large`	The JSON body exceeds 5 MiB.
`429`	`rate_limit_exceeded`, `insufficient_quota`	Per-minute request limit or daily token quota reached; honor `Retry-After`.

Authentication and scope are checked before the body is validated, so a JSON body with invalid fields and a missing key returns 401, not 400. A body that is not valid JSON is rejected with 400 before authentication (evaluation order). All errors use the standard envelope described in Errors.

Start a crawl​

Request​

Response​

Seed URL rules​

Domain allowlist​

How the crawl proceeds​

robots.txt​

Check the crawl status​

Crawled documents​

Re-crawling​

Errors​