robots.txt
robots.txt expresses website crawling preferences to cooperating crawlers. It is not access control.
In this section
- What is robots.txt
- Check details
- Common issues
- Platform guides: WordPress · Shopify · Joomla
Quick facts
- Location:
https://yoursite.com/robots.txt - Format: Plain text with
User-agent,Allow,Disallow, andSitemapdirectives - Specification: robotstxt.org
Use this reference
robots.txt communicates crawl preferences to cooperating crawlers. Flowpane observes the root /robots.txt address; it does not assume that the response is a physical file rather than CMS-generated content.
Start with What is robots.txt for purpose, Check details for the assessment, and Common issues for follow-up. robots.txt is public crawl guidance, not authentication or a way to keep confidential content private. Blocking crawling is also different from requesting noindex.
Decide which crawler groups and paths should be crawlable. A broad Disallow can affect many URLs; a narrow rule should reflect the actual path and intended crawler. Use Review for supported quick corrections and Workbench for full content and drafts. Saving a proposal is separate from publication.