Plugin searches should force to lowercase

[jonoaldersonwp]

Plugin searches like ​https://wordpress.org/plugins/search/Cats/ support mixed-case URLs.

The search form submission/handling process should return a lowercase URL, and, direct requests to mixed-cased URLs here should be forced to lowercase via a 301 redirect.

[dd32]

Changing this would be of detriment to end-users, and complicated for non-ascii languages

[jonoaldersonwp]

If you're concerned about the input box and H1 contents, then we could add some additional complexity to the ticket to address - we could de-couple the search query and how it's (re)presented to the user from the slug; maintaining the former, and forcing the latter to lowercase.

[edo888]

The search pages have "noindex,follow", does it really matter in terms of SEO or this is just for esthetics?

Also, why this is a high priority task? Which criterion is used to set a priority in the meta trac?

Thanks!

[jonoaldersonwp]

Yes, it matters. We have millions of pages on a site which is still profoundly unhealthy. We have hundreds of thousands of URLs which Google hasn't even evaluated yet, because crawling the site is so resource-intensive, and so poorly optimized. Every single individual URL which we can de-duplicate and bring up to something resembling best practice claws back crawl budget and resource for the URLs which matter.

[edo888]

Good point, thanks!

[dd32]

Given we already have this in robots.txt, is there any real reason why we shouldn't just add /plugins/search to it as well?

User-agent: *
Disallow: /search

I understand it's not the ideal option, but it seems pointless to have search engines access potentially millions of URLs that are intentionally noindexed.

This is not a high priority ticket.

[jonoaldersonwp]

Yeah, that's better than nothing; let's do that, and unpick the damages in some theoretical future.

[dd32]

In 11722:

Robots: Disallow /plugins/search/* for now, as they're all noindexed.

This should discourage search engines from hitting endless noindexes & affecting their rate limits.

See #5323

[jonoaldersonwp]

Worth noting that this might have some unexpected consequences; I know a number of sites/plugins/etc link directly to these types of URLs, in a way that currently assists discovery (and likely pass equity, despite the noindex status).

Blocking access to and visibility of that completely will change the balance of power in the ecosystem; we should expect potential grumblings as plugin installs decline in some cases.

[dd32]

Replying to jonoaldersonwp:

Worth noting that this might have some unexpected consequences; I know a number of sites/plugins/etc link directly to these types of URLs, in a way that currently assists discovery (and likely pass equity, despite the noindex status).

Hmm.. Noted, I can think of a few cases where that might be the case.

Let's see if we see any effective change from this, and re-evaluate it if needed.

[jonoaldersonwp]

Sounds like a plan!

[dd32]

In 11979:

WordPress.org Robots.txt: Correctly limit /plugins/search/* crawl limiting.

In [11722] I added blocking for the plugins/search endpoint, however, I did so by not including a trailing slash.
The result of this is that the rule has blocked access to plugins whose slugs begin with search (i.e. /plugins/search-*/).

By adding a trailing slash here, it correctly applies the Disallow rule to /plugins/search/* instead.

Props digamberpradhan for noticing/reporting the issue.
See #5323.
Fixes #6413.