Change meta robots directive on systems

[jonoaldersonwp]

At the moment, systems URLs (such as ​https://make.wordpress.org/systems/, ​https://make.wordpress.org/systems/2019/10/01/redirect-downloads-wordpress-org-to-wordpress-org-download-currently/) output a meta robots tag with a value of noindex, nofollow.

This should be altered to noindex, follow.

[Otto42]

That's not a tag which is specific to the systems blog, it is simply using the WordPress core "Discourage search engines from indexing this site" setting.

Which seems correct, as we don't want search engines indexing that site. Or looking at it in any way.

[jonoaldersonwp]

That tag/setting won't stop them looking at it. It prevents them from indexing it, and it prevents them from then following any links (though as of ~Oct, the nofollow directive in this case, as far as Google is concerned, is actually just a hint, rather than a directive).

If we want to stop them from looking at it, it needs blocking in robots.txt.

Given that isn't the case, we _do_ want them to follow links (and flow pagerank) _out_ of those pages back into other areas of the WP ecosystem.

If there's truly sensitive stuff here which we don't want crawling at all, it shouldn't be public.

[diddledani]

Replace robots noindex,nofollow with noindex,follow

[diddledani]

This patch (4817.diff) will require an admin to flip the option in the admin so that the site is considered public - do not discourage search engines. The reason the admin toggle needs to be flipped is because I am using wp_no_robots which will check the option value and continue using the current behaviour if it is still set to non-public. Setting it to public will tell wp_no_robots to change behaviour to allow follow but retain noindex.

[diddledani]

An alternative implementation that doesn't require the admin to change the site visibility/search engine option

[dd32]

Is this something specific to the make/systems site, or is this a generic suggestion that should be made to all WordPress.org sites if using the discourage search engine option?

If the latter, is this something that should be considered for change in WordPress core? or is nofollow probably the intended value for the vast majority of cases?

Any private-private site on WordPress.org is protected using a login plugin, and the make/systems site isn't designed to be hidden (We have private posts for anything potentially sensitive) just not encouraged to appear in search results.

[dd32]

Replying to dd32:

Is this something specific to the make/systems site, or is this a generic suggestion that should be made to all WordPress.org sites if using the discourage search engine option?

Example of why I'm asking; the default core value is also the reason behind #4807

[jonoaldersonwp]

We updated the "Discourage search engines" behaviour in core relatively recently, to output "noindex, follow". Are we running a recent enough version to have picked that up?

[dd32]

Replying to jonoaldersonwp:

We updated the "Discourage search engines" behaviour in core relatively recently, to output "noindex, follow". Are we running a recent enough version to have picked that up?

We are, but that change doesn't do what you think it does.. ​https://core.trac.wordpress.org/ticket/43590

wp_no_robots() still outputs noindex,nofollow unless the option "Discourage search engines" is disabled where it changes to noindex,follow. But if the option is disabled, wp_no_robots() isn't called by default, except on embeds and ?replytocom= links where it's called.

[jonoaldersonwp]

Ah, gotcha. Thanks!

[dd32]

r16042-dotorg