robots.txt check details

Flowpane reviews the public robots.txt for readable directives and findings that may affect your intended crawler policy.

Review the crawler groups, Allow and Disallow rules and Sitemap declarations alongside the source content. Confirm publisher intent before changing access preferences.

A missing file, an empty response and blocked access are different outcomes. Open the finding explanation rather than treating each as the same problem. A proposed starter does not establish your intended policy or publish changes.

See Common issues for remediation guidance.

Interpreting the result

Present means the response was obtained, not that every declaration is correct. Missing means it was not found at the checked location. Blocked or Error prevents a dependable current content assessment; read the explanation before drafting a replacement.

A rule disallowing /admin/ asks matching crawlers not to crawl that path. Protect an administration area with proper access controls as well.

An Editor or Admin on an eligible plan can prepare a draft; public publication requires the appropriate authority. After publishing, run a new check and compare the resulting observation time. See troubleshooting.