Plugins with 'noindex' rules should be excluded from XML sitemaps

[jonoaldersonwp]

Plugins such as ​https://wordpress.org/plugins/narnoo-shortcodes/, which output a meta robots tag which includes a noindex value, should be removed/excluded from XML sitemaps (e.g., ​https://wordpress.org/plugins/sitemap-index-1.xml
)

[Otto42]

AFAIK, we don't have any sitemaps on w.org.

There is a feature in Jetpack that can generate sitemaps, but that should be turned off on all the sites by default? It might have been on for some reason earlier, but appears to be off now.

Closing this since there isn't a sitemap, but if we decide to make one happen, then things like this should be considered in a more overarching ticket.

[dd32]

In 9506:

Plugin Directory: Skip outdated plugins from being included in the Jetpack Sitemaps.

This may cause OOM fatals due to the usage of get_post() in Template::is_plugin_outdated() which may exhaust the in-memory object-cache cache.

Fixes #5025.

[dd32]

This may cause OOM fatals due to the usage of get_post() in Template::is_plugin_outdated() which may exhaust the in-memory object-cache cache.

As expected:

Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes)
Timestamp: Mon, 09 Mar 2020 01:11:28 +0000 (1583716288)
File: object-cache.php
Line: 234
Filter stack: jp_sitemap_cron_hook, jetpack_sitemap_skip_post
Current filter: jetpack_sitemap_skip_post
- attached to priority 10: WordPressdotorg\Plugin_Directory\Plugin_Directory->jetpack_sitemap_skip_post

I guess we'll have to clear the in-memory cache occasionally

[dd32]

In 9650:

Plugin Directory: Clear memcache/db in-memory caches during XML Sitemap generation.

Fixes #5025.