How Flowpane works with WordPress

WordPress builds most of its machine-readable output on request. robots.txt, the XML sitemap and each page's indexing directives can come from WordPress core, an SEO plugin, the theme, a file on disk or a cache in front of the Site. Flowpane brings WordPress configuration and publicly observed behaviour together, helping teams identify inconsistencies across robots directives, sitemaps and indexing signals and correct them with evidence. This page explains how each output is produced, how to confirm which component owns it, and what the Flowpane connector adds.

For the general model, read How Flowpane works with your CMS.

robots.txt: physical file or virtual response

WordPress has two ways to answer /robots.txt:

Request for /robots.txt
 |
 +-- Is there a robots.txt file in the WordPress root directory?
 |     yes -> The web server returns that file. WordPress does not run,
 |            so WordPress settings and plugin robots.txt options that
 |            change the virtual response have no effect on it.
 |
 +-- no  -> WordPress generates a virtual response:
            core rules, then every plugin that adds to it
            (core sitemaps and SEO plugins usually add a Sitemap line).
 |
 +-- Either answer can then be cached by a page cache, host or CDN.

A typical virtual response from a WordPress Site with core sitemaps enabled looks like this (illustrative, example.com):

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

SEO plugins add robots.txt editors that work in one of two ways: writing a physical file, or adjusting the virtual response. When both exist, the physical file is what crawlers receive, which is why an edit made in a plugin can appear to have no public effect.

Confirm which one your Site serves

curl -sI https://example.com/robots.txt
curl -s https://example.com/robots.txt

Compare the body with the robots.txt settings in WordPress and your SEO plugin. If they differ, look for a robots.txt file in the WordPress root directory and for a cache in front of the Site. Response headers are a useful first signal: a static file is often served with Last-Modified and ETag from the web server, while a generated response usually is not. On a connected Site, the connector reports which of the two WordPress is serving.

XML sitemaps

Two producers can coexist by accident: an old physical sitemap.xml beside a plugin index, or a Sitemap: line in a physical robots.txt that still names an address the Site no longer serves. Flowpane's public checks discover the sitemaps crawlers can find and report each one with its own evidence.

curl -s https://example.com/wp-sitemap.xml | head -n 20
curl -s https://example.com/sitemap_index.xml | head -n 20

Page indexing directives

A page's <meta name="robots"> tag reflects several sources. WordPress builds it through its robots API, and SEO plugins and themes contribute through the same API or their own output:

• Search engine visibility (Settings → Reading → "Discourage search engines from indexing this site") adds noindex to every page.
• SEO plugin settings add per-post, per-term and per-content-type noindex or nofollow.
• The theme or other plugins can add directives through the same API, or print their own tag.
• The host or CDN can add an X-Robots-Tag header that sits outside the page HTML.

curl -s https://example.com/sample-page/ | grep -i 'name="robots"'
curl -sI https://example.com/sample-page/ | grep -i 'x-robots-tag'

The site-wide visibility switch is the most common source of unexpected noindex on a live Site, usually after a staging copy becomes production.

Caching and delivery

Page caches, host caches and CDNs can store robots.txt, sitemap XML and page HTML. After a change in WordPress, the public response may stay the same until the cache expires or is purged. Redirects between www and the bare domain, or from http to https, also decide which copy crawlers reach. Flowpane records the response crawlers receive, including the redirect path, so a cached or redirected answer is visible as such.

What Flowpane observes on every WordPress Site

Without a connection, Flowpane checks the public robots.txt, the sitemaps it can discover, the governance files and sampled pages, and records Results, Changes and Findings from what the public web serves. This is identical for every CMS.

What the connector adds

The connector reads WordPress configuration and sends counts, addresses and settings. Posts, files, sitemaps and SEO plugin settings remain under your team's control. See Source activity, health and change summaries.

Reported and verified on WordPress

Source evidence explains what WordPress intends. The public check decides what crawlers receive. Flowpane shows both, each with its own date:

• If your SEO plugin says a page is noindex and the public page agrees, the Result is consistent.
• If the plugin says index but the public page carries noindex, another source is adding it. The visibility switch, the theme or a header are the usual causes.
• If WordPress reports a robots.txt change but the public file is unchanged, a physical file or a cache is serving the response.

What managed changes permit on WordPress

A connection provides observation and reporting. Where Managed changes is enabled for the Site, a person with the right role can approve:

• Page directives: add noindex or nofollow to a specific page through WordPress's own robots API. Flowpane's directives sit alongside those already set by WordPress, the theme or other plugins, so protections such as the site-wide visibility setting stay in force. Directives are tied to the connection and end when the Site is disconnected.
• The Flowpane-managed sitemap: Flowpane's own sitemap, served at /flowpane-sitemap.xml with pretty permalinks or /?flowpane_sitemap=index with plain permalinks. It runs alongside your existing sitemaps from core, Yoast, Rank Math or any other tool.

Each action is approved in Flowpane, applied by the connector and then checked on the public Site. robots.txt stays with whichever tool produces it today; Flowpane reports and verifies it.

Worked examples

A staging setting reaches production. A Site is copied from staging with "Discourage search engines" still on. The connector reports the setting and sends a priority event. Flowpane checks the public Site and records the noindex directive that crawlers now receive on its pages. The source evidence names the setting to change, so the fix is a single switch rather than a page-by-page search.

A forgotten physical file. An SEO plugin is configured to add its sitemap to robots.txt, but the public robots.txt never shows the line. The connector reports that robots.txt comes from a physical file. Removing or updating that file, after agreeing ownership with the maintainer, lets the virtual response through. The next public check records the Change.

A cached page after a policy change. An editor sets a page to noindex in the SEO plugin. The connector reports the new setting, but the public check still sees the old HTML from a page cache. Flowpane records the Change when the public page actually changes, and in the meantime the evidence points to the cache as the item to clear.

Environments

Supported WordPress, PHP and database versions are listed in WordPress compatibility. For plugin-specific guidance, see Yoast and Rank Math.

Flowpane for WordPress · robots.txt on WordPress · Managed changes