Crawl URLs and sitemaps
Add help-center pages and other web content by sitemap or start URL, with include and exclude patterns, page limits and recrawl schedules.
Use a crawl source for content that isn't in your platform, such as a help center or a blog.
Add a source
In Knowledge → Sources → Add source:
| Field | Notes |
|---|---|
| Kind | Sitemap (a sitemap URL) or URL (a start page; links on the same site are followed) |
| Include / exclude patterns | Globs such as /help/** or **/archive/** |
| Max pages | Up to the pages left on your plan |
| Recrawl every | 6, 12, 24, 72 or 168 hours |
The crawler always respects robots.txt. It identifies itself with the ShopAssist bot user agent and crawls from the egress IPs; see ShopAssist bot.
Other domains
Pages on a domain that isn't one of your store's domains need approval on first import (review queue). Policy pages found by a crawl always need approval.
Large changes are held
If a recrawl would change more than 30 % of a source's approved documents at once, all changes of that run wait for review and the dashboard shows knowledge_mass_change. This catches site redesigns or error pages.
Errors
URLs must be public HTTPS on the standard port. Private or local addresses are refused (url_not_public, port_not_allowed).
Last updated on