Robots.txt
Manage the disallow rules in your storefront's robots.txt file to control search engine crawler access.
The Robots.txt page lets you manage the Disallow rules that appear in your storefront's robots.txt file. Search engine crawlers read this file to determine which paths on your site they should not index. Each rule you add here becomes a Disallow: line in the generated robots.txt.
The storefront dynamically generates robots.txt on every request. It outputs a User-agent: * header, your configured disallow rules, and a Sitemap: reference pointing to your storefront's sitemap.xml. If the storefront is running on a temporary domain, all paths are blocked with Disallow: / to prevent indexing of staging or test sites.
View Robot Rules
- From your admin, go to Settings > Domain Tools > Robots.txt (labeled "Robot Rule" in the page header).
- The page displays a paginated grid of all configured disallow paths.
Grid Columns
| Column | Description |
|---|---|
| Path | The URL path to disallow. This value is output as-is after Disallow: in the generated robots.txt. Use paths like /admin/ or /checkout/ to block specific sections from crawlers. |
| Action | Menu with one option: Delete. |
Add Robot Rule
- Click Add Robot Rule in the page header.
- In the dialog, enter the Path to disallow (e.g.,
/private/). - Click Save.
The path cannot be empty. The rule takes effect on the next request to your storefront's robots.txt.
Delete Robot Rule
- Click the Action menu on any rule row and select Delete.
- Confirm the deletion in the dialog.
The disallow line is removed from the generated robots.txt. Both add and delete actions are logged in the Activity Log.
Generated Output
The storefront serves robots.txt at the root of your domain (e.g., https://yourdomain.com/robots.txt). The generated output follows this format:
User-agent: *
Disallow: /path-1/
Disallow: /path-2/
Sitemap: https://yourdomain.com/sitemap.xmlIf the storefront is accessed via a temporary domain (used for staging or testing), the output blocks all crawling:
User-agent: *
Disallow: /Associations
Print Store
| Area | Effect |
|---|---|
| robots.txt endpoint | The storefront has an HTTP handler (RobotsHandler) registered for robots.txt that dynamically generates the file from your configured rules on every request. |
| Sitemap reference | The generated robots.txt automatically includes a Sitemap: directive pointing to your storefront's sitemap.xml, helping crawlers discover your site structure. |
| Temporary domain blocking | Storefronts on temporary domains automatically return Disallow: / for all paths, preventing search engines from indexing staging or test sites. |
Related Pages
- Sitemap — the sitemap referenced automatically in your robots.txt output
- Redirects — manage URL redirects that work alongside crawler rules
- Site Bindings — connect a custom domain, which affects temporary-domain crawl blocking