Common robots.txt issues

Blanket disallow

An effective Disallow: / asks the matched crawler not to fetch the Site. It may be intentional and is not access control.

Confirm publisher intent before changing it. Flowpane does not automatically reverse a blanket restriction.

Missing sitemap directive

Without a Sitemap: line, search engines must discover your sitemap on their own.

Add a Sitemap: directive using the public sitemap URL confirmed in your Site or CMS configuration.

Mixed protocols

Mixed origin/protocol declarations can disagree with the effective Site and deserve review.

Preserve an explicit HTTP or custom-port deployment when it is authoritative; otherwise update the declaration to the verified effective URL.

File not found

A 404 is a genuine missing result, distinct from a successful empty file, blocked response or acquisition error.

Review any starter draft against your intended crawler policy. Confirm the real sitemap URL and any platform-specific restrictions before publishing.

Resolve the cause, then verify

First compare the finding with the obtained Current content. If the expected text URL returns a theme page, challenge or redirect, ask the responsible host or CMS maintainer to correct that response. A browser-visible page does not establish that the resource was readable to Flowpane.

Decide which crawler groups and paths should be crawlable. A broad Disallow can affect many URLs; a narrow rule should reflect the actual path and intended crawler.

Prepare a focused correction, check authored details and use your authorised publication process. robots.txt is public crawl guidance, not authentication or a way to keep confidential content private. Blocking crawling is also different from requesting noindex. A saved draft or dismissed notice is not proof of remediation; repeat the public check afterwards.