# Crawl URLs and sitemaps

> Add help-center pages and other web content by sitemap or start URL, with include and exclude patterns, page limits and recrawl schedules.

Source: https://shopassist.coec.dk/docs/knowledge-base/crawl-urls-and-sitemaps · Updated 2026-10-01

Use a crawl source for content that isn't in your platform, such as a help center or a blog.

## Add a source [#add]

In **Knowledge → Sources → Add source**:

| Field                      | Notes                                                                                      |
| -------------------------- | ------------------------------------------------------------------------------------------ |
| Kind                       | **Sitemap** (a sitemap URL) or **URL** (a start page; links on the same site are followed) |
| Include / exclude patterns | Globs such as `/help/**` or `**/archive/**`                                                |
| Max pages                  | Up to the pages left on your plan                                                          |
| Recrawl every              | 6, 12, 24, 72 or 168 hours                                                                 |

The crawler always respects `robots.txt`. It identifies itself with the ShopAssist bot user agent and crawls from the [egress IPs](/docs/security/egress-ips); see [ShopAssist bot](/docs/bot).

## Other domains [#third-party]

Pages on a domain that isn't one of your store's domains need approval on first import ([review queue](/docs/knowledge-base/review-queue)). Policy pages found by a crawl always need approval.

## Large changes are held [#mass-change]

If a recrawl would change more than 30 % of a source's approved documents at once, all changes of that run wait for review and the dashboard shows [`knowledge_mass_change`](/docs/troubleshooting/sync-issues#knowledge_mass_change). This catches site redesigns or error pages.

## Errors [#errors]

URLs must be public HTTPS on the standard port. Private or local addresses are refused (`url_not_public`, `port_not_allowed`).