How Flowpane works with Drupal

Drupal core deliberately does little on its own for search engines: robots.txt arrives as a file, sitemaps, metatags and automatic URL aliases come from contributed modules, and caching can sit in Drupal, the web server or a CDN. That flexibility means two Drupal Sites can answer the same question in entirely different ways. Flowpane examines what each Site actually delivers, and the module adds Drupal's own account of which component is responsible, so a public result can always be traced to its source.

For the general model, read How Flowpane works with your CMS.

robots.txt: a scaffolded file or a module

Drupal ships a physical robots.txt in the web root, placed there by Drupal's Composer scaffolding. Some Sites replace it with the RobotsTxt module, which serves the rules from Drupal's configuration instead:

Request for /robots.txt
 |
 +-- Web server or hosting rule answers?        -> that response is served
 +-- robots.txt file in the web root?           -> the file is served,
 |                                                 whatever the module says
 +-- no file, RobotsTxt module enabled?         -> the module's rules are served
 |
 +-- A CDN or reverse proxy can serve an older copy.

Because the file comes from scaffolding, a routine composer update can put Drupal's default file back over local edits unless the project's scaffold settings exclude it. The module reports whether the file exists, whether scaffolding manages it, and whether the RobotsTxt module holds its own rules, so a surprising public response can be explained at once.

curl -s https://example.com/robots.txt

XML sitemaps come from modules

Drupal core has no XML sitemap generator. The two common modules work differently:

The module reports which sitemap modules are enabled and how many sitemaps or variants each defines. Flowpane then reads each sitemap from the public web and parses it independently.

Addresses and URL aliases

Every Drupal page has an internal path, such as /node/42, and usually a friendlier URL alias, such as /about-us, created by hand or automatically by Pathauto. Both normally reach the same content, so the alias a sitemap lists and the address a page declares as canonical matter. The Redirect module can send old aliases to new ones when an alias changes.

The module reports each published node translation with its internal route and its alias, so Flowpane can compare the addresses Drupal intends with the addresses sitemaps list and the public Site serves.

Languages and translations

On a multilingual Site, each translation usually has its own language prefix or domain and its own alias. The module reports the Site's languages and each published translation separately, so a page that is published in one language and not another is visible as exactly that.

Indexing directives come from Metatag

Drupal core does not print a robots meta tag on content. Where a Site uses the Metatag module, its defaults, content type settings and per-page overrides decide whether a page carries noindex or nofollow; themes and the web server can add their own directives too. The module reports whether Metatag is present, so Flowpane knows where indexing intent is set, and the public page remains the final answer:

curl -s https://example.com/about-us | grep -i 'name="robots"'
curl -sI https://example.com/about-us | grep -i 'x-robots-tag'

Caching and delivery

Drupal's Internal Page Cache serves stored pages to anonymous visitors, the Dynamic Page Cache stores page fragments, and many Sites add a reverse proxy or CDN in front. Any of them can serve an older copy of a page or sitemap after a change. The module reports which Drupal caches are active; Flowpane records the response crawlers actually receive, including redirects.

What Flowpane observes on every Drupal Site

Without a connection, Flowpane checks the public robots.txt, the sitemaps it can discover, the governance files and sampled pages, and records Results, Changes and Findings from what the public web serves. This is identical for every CMS.

What the module adds

The module sends counts, addresses, dates and fingerprints of configuration; your content stays on your Site.

Reported and verified on Drupal

Flowpane shows the module's report and the public observation separately, each with its own date. When the two disagree, the gap is the useful part:

• If the RobotsTxt module holds rules but the public file differs, a physical robots.txt in the web root is answering first.
• If a published translation is missing from the sitemap, the sitemap module's settings for that language or content type are the place to look.
• If Metatag says index but the public page carries noindex, the theme, another module or a header is adding it.

Worked examples

A Composer update restores the default robots.txt. A team customises robots.txt, then a routine composer update restores Drupal's default file through scaffolding. The module reports that the file changed and that scaffolding manages it; the public check records the Change with before and after content, and the fix is a one-line scaffold setting rather than repeated hand edits.

An alias changes. An editor renames a page and Pathauto generates a new alias. The module reports the new alias for that translation, Flowpane checks the public Site, and the evidence shows whether the Redirect module now sends the old address to the new one.

Two sources of rules. A Site enables the RobotsTxt module but never removes the physical file. Changes made in the module's settings have no public effect; the module's report shows both sources at once, so the team can agree which one should own robots.txt.

Environments

Requirements are listed on Flowpane for Drupal.

Flowpane for Drupal · Managed changes · Extensions Hub