Dedicated robots.txt file for translate.wordpress.org

[jonoaldersonwp]

This consumes huge amounts of crawl budget, for relatively little return. We'd like to block crawling of it entirely, via robots.txt.

At the moment, it shares a robots.txt with other wordpress.org domains, which makes this impossible.

Can we give it a dedicated robots.txt file, which is separate from other sites, with the following contents:

User-agent: *
Disallow: /*
Noindex: /*
Allow: /$

NB: We'll need to be absolutely certain that this is a standalone file, and doesn't bleed through to any other WP domains/contexts, or we'll cause the end of the world.

If/when this is complete, the ?filter rule can be removed from the shared/global robots.txt file.

[ocean90]

There are probably a few pages which still should be indexed like /stats, /consistency or each /locale/$locale.

We'd like to block crawling of it entirely

Just out of curiosity, who is "we"?

[jonoaldersonwp]

Happy to add a small number of whitelisted pages. In this case, "we" is me and Joost.

[dd32]

In 9070:

GlotPress: Add a handler to allow robots.txt to be served by WordPress on translate.wordpress.org.

See #4559.

[dd32]

In 9071:

Add a mu-plugin which contains the customised WordPress.org robots.txt content.

The intention is that we can switch from serving a static robots.txt file to serving a customised one from WordPress.

See #4559.

[dd32]

After some discussion with @tellyworth, I think the best route forward here is to switch to using the WordPress generated robots.txt and remove the static robots.txt that's currently in place.

That has some downsides, one of them being the potential unknowns (currently) of there being a domain that we're not thinking of that is currently relying upon the static file.. But assuming we can work around that, it seems like moving is the best option here.

The above commits implement the existing robots.txt file and the request here into something that is usable.

Currently:

• ​https://*/robots.txt is still using the static file
• The WordPress generated file can be accessed like so: ​https://wordpress.org/?robots=1 or ​https://de.wordpress.org/?robots=1
• For translate.wordpress.org, it can be found here: ​https://translate.wordpress.org/?robots=1&gp_route=robots.txt

The Robots file for translate.wordpress.org..

• Drops the noindex: declaration as it'll be ​no longer supported by Google as of September.
• Adds the Global Stats, Consitency Report, locale index, the Locale Indexes and Locale Plugin/Theme stats to the Allow ruleset.

I couldn't find many other pages worth indexing on translate.wordpress.org, we really don't need to index every project in every locale or all the strings, so having Search engines request those URLs isn't doing us any benefit.

[dd32]

In 9072:

Robots.txt: Don't link to the Themes sitemap.xml, as there's a server-level redirect in place breaking it.

See #4559.

[dd32]

Replying to dd32:

In 9072:

Robots.txt: Don't link to the Themes sitemap.xml, as there's a server-level redirect in place breaking it.

See #4559.

For reference, ​https://make.wordpress.org/systems/2019/07/24/remove-trailingslashit-rule-for-theme-directory-uris/ to remove the redirect that breaks it.

[jonoaldersonwp]

Any movement on this? :)

[ocean90]

#4186 was marked as a duplicate.

[dd32]

In 9350:

Make: Ensure that make.wordpress.org/robots.txt works without a statically defined file.

See #4559.