Hide Your Website From Search Engines: Block Crawler Access Guide
Excluding your site from search engines effectively controls its visibility while keeping it accessible to direct visitors. Here's how to manage your site's search engine visibility:
Blocking All Search Engines
To hide your entire website from search results:
- Go to Settings > Crawlers
- Check "Block search engine crawlers"
This adds a robots.txt rule preventing search engines from indexing your site.
Excluding Specific Pages
You can hide individual pages using two methods:
Page Settings Method:
- Open Pages panel
- Click the settings gear icon for the page
- Go to SEO tab
- Toggle "Hide page from search results"
Benefits:
- Available on all plans
- No coding required
- Removes page from sitemap
- Built into page settings
Code Injection Method: Add this code to the page header:
<meta name="robots" content="noindex" />
Benefits:
- Works on homepages
- Keeps page in sitemap
- Available for advanced customization
Important Notes:
- Hiding a collection page also hides all items within it
- Individual collection items cannot be hidden separately
- Index page settings don't affect secondary pages
- These methods only block search engines, not site search
- Pages remain accessible via direct links
To ensure complete coverage, apply the noindex tag to both main and secondary pages if hiding an entire section.
Remember: While hidden from search engines, your site remains accessible to visitors who have the direct URL.
For AI crawler exclusion, use the dedicated AI scan exclusion settings in your site configuration.