Stale themes should be noindex'd

[jonoaldersonwp]

As per #4088, themes which haven't been updated in over two years should be noindex'd.

E.g., ​https://wordpress.org/themes/destro/

[Otto42]

Why 2 years? Let's be honest, the 2 years thing was a number I plucked out of thin air almost 10 years ago when the directories were somewhat different.

Would a different number make more sense?

[jonoaldersonwp]

Ah, good question! I just piggybacked on the number in the notification banner, but would be happy with any sensible value.

How long does it take a theme to (likely) go 'stale'? I wonder if a year isn't long enough, and that we might end up impacting the visibility of valid-but-outdated themes? I also wonder if that time-to-stale might change radically over the course of adoption of Gutenberg, and also, decrease over time as block support proliferates...

Are there other opinions/policies on this?

Noindex'ing them also removes them from internal search, right? So we should probably be careful to avoid false-positives.

[dd32]

noindex doesn't remove it from the search, but yes, the themes search includes age as one of it's requirements (bypassing that for a exact match).

The only thing that this will affect is ranking in search engines, something I'm willing to ditch given we don't promote such themes.

Questions on if 2 years is the right flag to use here is valid, but since we're already using that magic number elsewhere for themes we should probably figure that out separately.

[dd32]

In 9471:

Theme Directory: Output a noindex robots tag for themes marked as potentially outdated.

Fixes #4994.

[jonoaldersonwp]

Found a few examples which I think should be noindex'd, but don't appear to be.
Any idea why these are slipping through the net?

​https://wordpress.org/themes/guruq/
​https://wordpress.org/themes/chinese-love/
​https://en-gb.wordpress.org/themes/portfilo/

[ocean90]

Hmm. maybe check again? All three themes have the <meta name='robots' content='noindex,follow' /> tag.

[jonoaldersonwp]

Interesting. I get radically different markup / meta when logged in vs logged out. Is that intentional?

[dd32]

Looks like something to do with caching.

$ curl -is https://wordpress.org/themes/guruq/ | grep '\(robots\|x-nc\)'
x-nc: MISS ord 1
<meta name='robots' content='noindex,follow' />

$ curl -is https://wordpress.org/themes/chinese-love/ | grep '\(robots\|x-nc\)'
x-nc: MISS ord 1
<meta name='robots' content='noindex,follow' />

$ curl -is https://en-gb.wordpress.org/themes/portfilo/ | grep '\(robots\|x-nc\)'
x-nc: MISS ord 2
$ curl -is https://en-gb.wordpress.org/themes/portfilo/ | grep '\(robots\|x-nc\)'
x-nc: MISS ord 2
<meta name='robots' content='noindex,follow' />

It's not nginx caching, so I'm not sure what to make of that at first.

[dd32]

Yup, looks like it's some kind of caching, the $theme data isn't available when the header functions run:

edit: It's even weirder. is_single() returns true, but get_post() returns null.. 🤷

$ curl -is https://en-gb.wordpress.org/themes/portfilo/ | grep '\(<meta\|x-nc\)'
x-nc: MISS ord 1
<meta charset="utf-8" />
<meta property="fb:page_id" content="6427302910" />
<meta name="google-site-verification" content="7VWES_-rcHBcmaQis9mSYamPfNwE03f4vyTj4pfuAw0" />
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="referrer" content="always">
<meta name="generator" content="WordPress 5.4-beta3-47420" />
<meta name="description" content="Free WordPress Themes" />

Seconds later:
$ curl -is https://en-gb.wordpress.org/themes/portfilo/ | grep '\(<meta\|x-nc\)'
x-nc: MISS ord 1
<meta charset="utf-8" />
<meta property="fb:page_id" content="6427302910" />
<meta name="google-site-verification" content="7VWES_-rcHBcmaQis9mSYamPfNwE03f4vyTj4pfuAw0" />
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="referrer" content="always">
<meta name="generator" content="WordPress 5.4-beta3-47420" />
<meta property='og:title' content='Portfilo' />
<meta property='og:description' content='Portfilo is a responsive theme designed for businesses to help implement their online presence along with blog. With the help of recommended free plugins, you can showcase your portfolio and testimonials on the website. Theme provides customizer options to change look and feel of your website. Theme provides options to change logo, favicon, color scheme, tyapography, etc. It supports popular plugins like BuddyPress, WooCommerce and WPML. Portfilo theme is translatable so that you can easily localize it in the language you want.' />
<meta property='og:site_name' content='WordPress.org' />
<meta property='og:type' content='website' />
<meta property='og:url' content='https://en-gb.wordpress.org/themes/portfilo/' />
<meta property='og:image' content='https://i0.wp.com/themes.svn.wordpress.org/portfilo/1.0.18/screenshot.png' />
<meta name='twitter:card' content='summary_large_image'>
<meta name='twitter:site' content='@WordPress'>
<meta name='twitter:image' content='https://i0.wp.com/themes.svn.wordpress.org/portfilo/1.0.18/screenshot.png?w=560&amp;strip=all' />
<meta name='robots' content='noindex,follow' />
<meta name="description" content="Portfilo is a responsive theme designed for businesses to help implement their online presence along with blog. With the help of recommended free plugins, you can showcase your portfolio and testimonials on the website. Theme provides customizer options to change look and feel of your website. Theme provides options to change logo, favicon, color scheme,&hellip;" />

[jonoaldersonwp]

Yikes! How confident are you that this is only affecting logged-in users? Do we need to worry that this is sporadically the output for logged-out folks (inc Google)?

Either way, it'd be good to fix it for logged-in folks, too; some of those meta tags are important for accessibility, etc.

[dd32]

Replying to jonoaldersonwp:

Yikes! How confident are you that this is only affecting logged-in users? Do we need to worry that this is sporadically the output for logged-out folks (inc Google)?

Nah, affects everyone. I have no idea what's going on - Can't fix what I can't figure out.

[jonoaldersonwp]

Eek. How do we escalate this and get more eyes/brains on it?
That's going to have a pretty catastrophic effect on performance.

[dd32]

#5192 was marked as a duplicate.

[dd32]

In 9870:

Theme Directory: Switch to using get_queried_object() instead of get_post() due to get_post not returning correctly on the theme directory sometimes.

Also moves the outdated theme noindexing to it's own filter.

See #5173.
Fixes #4994.

[dd32]

In 9935:

Theme Directory: Don't unset the global $post variable which causes a lot of "weird" failures of functions on a cold cache.

This should resolve the long-standing "what is going on here" questions on the Theme Directory when it comes to the SEO oddities.

See r9949-dotorg (5 years ago).
See #4994.

[dd32]

In 9936:

Theme Directory: Swap back to using get_post() now that it works as expected.

See [9935].
Reverts parts of [9870].
See #5173, #4994.