Document indexing controls

[jonoaldersonwp]

We've made a lot of progress in introducing indexing controls throughout the wp.org ecosystem, and recently, much of this has been consolidated into a dedicated SEO plugin for the site(s).

Increasingly, the controls we have in place are intertwining and becoming more complex. In order to avoid unexpected side-effects and edge-cases (and to potential consolidate redundant/overlapping parts), we should document the indexing criteria in place(s), and maintain that documentation as we introduce more conditions. This will also help define criteria for automated testing.

I'm not sure where such documentation should live (in the wp.org SEO plugin itself, perhaps?), but I've taken a stab at gathering and categorising the existing behaviours that I can see from historical tickets. This also includes open tickets, for the sake of visibility and planning.

NB: We should also consider expanding the documentation later to include other SEO considerations (canonical URL behaviour, redirects, 404 management, etc).

What are the next steps here / who can own this?

---

Forums

• Threads with < 100 chars and no reply: #4283
• Stale topic tag indexes/archives: #4338
• Topic tag archives: #5338
• Empty plugin/theme support categories: #4384
• Closed (+ aged) threads: #4025
• Closed (+ aged + no reply) threads: #4027
• "All topics" view: #5215

Profiles

• Banned users: #4632
• Inactive/empty user profiles: #4754
• Plugin committer URLs: #4329
• Empty support profile subpages: #4548
• User subpages: #4769
• Empty profiles: #4714

Developer.wordpress.org

• Block editor search: #4540
• post_type requests: #4648

Plugins

• Stale plugin peripherals: #5168
• Thin plugin tags: #4543
• Search results: #3850
• No search results template: #3952
• Empty reviews archive: #4323
• Date archives: #5182

News

• Categories: #5068
• Author archives: #5075
• "Thank you" states: #5254

Themes

• Stale themes: #4994
• Stale themes: #5192
• Theme search templates: #4327
• Author archives: #4549
• Tag archives: #5255

Make

• Thin tag archives: #4687
• All tag archives: #4800
• Search results: #4188
• "Resolved" views: #5333
• Date archives: #4688
• Mentions: #4689
• Entire 'Systems' site: #4817
• Recent comments: #5334

Trac

• Changesets: #4168
• Revisions: #4169
• Stale tickets: #4385
• Various: #4684
• Ticket attachments: #4129

Codex

• "Special contributions" codex pages: #4126
• File pages: #4373
• User pages: #4127

Rosetta

• Root date archives: #5187
• Tag archives: #5188
• Search: #5189

Misc

• Sync noindex to XML sitemap(s): #5025
• Jobs search: #5002
• Job categories: #5001
• Ideas search: #4128
• Login/register pages: #4807
• 5FF search: #5114

x-robots

• All feeds: #4336
• Theme tag feeds: #4337
• Ideas feeds: #5070
• wp-json: #5038

[dd32]

In 10385:

Open-source a bunch of redirects that were in sunrise.php (and a handful of similarly related redirects that didn't exist).

This will hopefully allow simpler recording of some of the generic redirects that are active, and may allow others to easier add/patch them.

See #5345.