Upload domains

Crawl a domain into the project knowledge base

POST /organizations/{organization_id}/projects/{project_id}/sources/domains

cURL

curl -X POST "https://backend.botcadence.com/organizations/ORG_ID/projects/PROJECT_ID/sources/domains" \
  -H "X-API-Key: bot_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "https://example.com",
    "crawling_strategy": "deep_crawl",
    "max_pages": 50,
    "max_depth": 3,
    "include_patterns": ["/docs/.*"],
    "exclude_patterns": ["/blog/.*"],
    "frequency_days": 30,
    "crawler": "crawl4ai"
  }'

const response = await fetch(
  `https://backend.botcadence.com/organizations/${organizationId}/projects/${projectId}/sources/domains`,
  {
    method: "POST",
    headers: {
      "X-API-Key": process.env.BOTCADENCE_API_KEY,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      domain: "https://example.com",
      crawling_strategy: "deep_crawl",
      max_pages: 50,
      max_depth: 3,
      include_patterns: ["/docs/.*"],
      exclude_patterns: ["/blog/.*"],
      frequency_days: 30,
      crawler: "crawl4ai",
    }),
  }
);

import os
import requests

response = requests.post(
    f"https://backend.botcadence.com/organizations/{organization_id}/projects/{project_id}/sources/domains",
    headers={
        "X-API-Key": os.environ["BOTCADENCE_API_KEY"],
        "Content-Type": "application/json",
    },
    json={
        "domain": "https://example.com",
        "crawling_strategy": "deep_crawl",
        "max_pages": 50,
        "max_depth": 3,
        "include_patterns": ["/docs/.*"],
        "exclude_patterns": ["/blog/.*"],
        "frequency_days": 30,
        "crawler": "crawl4ai",
    },
)

201

{
  "status": "success",
  "message": "Domain added successfully",
  "data": null,
  "status_code": 201
}

Adds a domain crawl source and queues discovery/indexing. Returns 201 Created.

Authentication

X-API-Key · string · required

Project API key for this organization and project. Format: bot_live_…

Path parameters

organization_id · string · required

Organization ID. Must match the API key.

project_id · string · required

Project ID. Must match the API key.

Body

domain · string · required

Domain or site root to crawl (for example https://example.com).

crawling_strategy · string · default: "deep_crawl"

sitemap or deep_crawl.

max_pages · integer · default: "50"

Maximum pages to crawl. Range: 1–1000.

max_depth · integer · default: "3"

Maximum crawl depth. Range: 1–10.

include_patterns · array

Whitelist regex/glob patterns; only matching URLs are crawled.

exclude_patterns · array

Patterns to exclude from the crawl.

frequency_days · integer · default: "30"

Automatic recrawl interval in days (1–365).

crawler · string · default: "crawl4ai"

Crawler engine: crawl4ai or apify.

Response

Success message confirming the domain crawl was queued. data is typically null.

Errors

401 · Unauthorized

Missing, invalid, or revoked API key.

403 · Forbidden

API key is not authorized, or link quota exceeded.

cURL

curl -X POST "https://backend.botcadence.com/organizations/ORG_ID/projects/PROJECT_ID/sources/domains" \
  -H "X-API-Key: bot_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "https://example.com",
    "crawling_strategy": "deep_crawl",
    "max_pages": 50,
    "max_depth": 3,
    "include_patterns": ["/docs/.*"],
    "exclude_patterns": ["/blog/.*"],
    "frequency_days": 30,
    "crawler": "crawl4ai"
  }'

const response = await fetch(
  `https://backend.botcadence.com/organizations/${organizationId}/projects/${projectId}/sources/domains`,
  {
    method: "POST",
    headers: {
      "X-API-Key": process.env.BOTCADENCE_API_KEY,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      domain: "https://example.com",
      crawling_strategy: "deep_crawl",
      max_pages: 50,
      max_depth: 3,
      include_patterns: ["/docs/.*"],
      exclude_patterns: ["/blog/.*"],
      frequency_days: 30,
      crawler: "crawl4ai",
    }),
  }
);

import os
import requests

response = requests.post(
    f"https://backend.botcadence.com/organizations/{organization_id}/projects/{project_id}/sources/domains",
    headers={
        "X-API-Key": os.environ["BOTCADENCE_API_KEY"],
        "Content-Type": "application/json",
    },
    json={
        "domain": "https://example.com",
        "crawling_strategy": "deep_crawl",
        "max_pages": 50,
        "max_depth": 3,
        "include_patterns": ["/docs/.*"],
        "exclude_patterns": ["/blog/.*"],
        "frequency_days": 30,
        "crawler": "crawl4ai",
    },
)

201

{
  "status": "success",
  "message": "Domain added successfully",
  "data": null,
  "status_code": 201
}