WordPress.org

Making WordPress.org

Opened 8 months ago

Closed 2 months ago

#4817 closed defect (fixed)

Change meta robots directive on systems

Reported by: jonoaldersonwp Owned by:
Milestone: Priority: lowest
Component: Make (Get Involved) / P2 Keywords: seo has-patch
Cc:

Description

At the moment, systems URLs (such as https://make.wordpress.org/systems/, https://make.wordpress.org/systems/2019/10/01/redirect-downloads-wordpress-org-to-wordpress-org-download-currently/) output a meta robots tag with a value of noindex, nofollow.

This should be altered to noindex, follow.

Attachments (2)

4817.diff (831 bytes) - added by diddledan 6 months ago.
Replace robots noindex,nofollow with noindex,follow
4817-alt.diff (937 bytes) - added by diddledan 6 months ago.
An alternative implementation that doesn't require the admin to change the site visibility/search engine option

Download all attachments as: .zip

Change History (12)

#1 @Otto42
8 months ago

That's not a tag which is specific to the systems blog, it is simply using the WordPress core "Discourage search engines from indexing this site" setting.

Which seems correct, as we don't want search engines indexing that site. Or looking at it in any way.

#2 @jonoaldersonwp
8 months ago

That tag/setting won't stop them looking at it. It prevents them from indexing it, and it prevents them from then following any links (though as of ~Oct, the nofollow directive in this case, as far as Google is concerned, is actually just a hint, rather than a directive).

If we want to stop them from looking at it, it needs blocking in robots.txt.

Given that isn't the case, we _do_ want them to follow links (and flow pagerank) _out_ of those pages back into other areas of the WP ecosystem.

If there's truly sensitive stuff here which we don't want crawling at all, it shouldn't be public.

@diddledan
6 months ago

Replace robots noindex,nofollow with noindex,follow

#3 @diddledan
6 months ago

  • Keywords has-patch added

#4 @diddledan
6 months ago

This patch (4817.diff) will require an admin to flip the option in the admin so that the site is considered public - do not discourage search engines. The reason the admin toggle needs to be flipped is because I am using wp_no_robots which will check the option value and continue using the current behaviour if it is still set to non-public. Setting it to public will tell wp_no_robots to change behaviour to allow follow but retain noindex.

Last edited 6 months ago by diddledan (previous) (diff)

@diddledan
6 months ago

An alternative implementation that doesn't require the admin to change the site visibility/search engine option

#5 follow-up: @dd32
5 months ago

Is this something specific to the make/systems site, or is this a generic suggestion that should be made to all WordPress.org sites if using the discourage search engine option?

If the latter, is this something that should be considered for change in WordPress core? or is nofollow probably the intended value for the vast majority of cases?

Any private-private site on WordPress.org is protected using a login plugin, and the make/systems site isn't designed to be hidden (We have private posts for anything potentially sensitive) just not encouraged to appear in search results.

#6 in reply to: ↑ 5 @dd32
5 months ago

Replying to dd32:

Is this something specific to the make/systems site, or is this a generic suggestion that should be made to all WordPress.org sites if using the discourage search engine option?

Example of why I'm asking; the default core value is also the reason behind #4807

#7 follow-up: @jonoaldersonwp
5 months ago

We updated the "Discourage search engines" behaviour in core relatively recently, to output "noindex, follow". Are we running a recent enough version to have picked that up?

#8 in reply to: ↑ 7 @dd32
5 months ago

Replying to jonoaldersonwp:

We updated the "Discourage search engines" behaviour in core relatively recently, to output "noindex, follow". Are we running a recent enough version to have picked that up?

We are, but that change doesn't do what you think it does.. https://core.trac.wordpress.org/ticket/43590

wp_no_robots() still outputs noindex,nofollow unless the option "Discourage search engines" is disabled where it changes to noindex,follow. But if the option is disabled, wp_no_robots() isn't called by default, except on embeds and ?replytocom= links where it's called.

#9 @jonoaldersonwp
5 months ago

Ah, gotcha. Thanks!

#10 @dd32
2 months ago

  • Resolution set to fixed
  • Status changed from new to closed

r16042-dotorg

Note: See TracTickets for help on using tickets.