Robots.txt blocking Search Engines from getting plugin descrpition

[digamberpradhan]

At the time of opening this ticket - July 19th 2022,
Our plugin Search with Typesense ​http://wordpress.org/plugins/search-with-typesense/
is being blocked for search engines.
That is when you type "Search with Typesense"
You will be shown "No information is available for this page" under the description of google search result - this is because the wordpress.org robots.txt is blocking all pages/urls with plugins/search

​https://wordpress.org/robots.txt

Other pages/plugins suffering the same fate:
​https://wordpress.org/plugins/search/search/
​https://wordpress.org/plugins/search-exclude/
​https://wordpress.org/plugins/search-in-place/
To name a few

[digamberpradhan]

Screenshot of Google Search Listing showing No information available

[dd32]

In 11979:

WordPress.org Robots.txt: Correctly limit /plugins/search/* crawl limiting.

In [11722] I added blocking for the plugins/search endpoint, however, I did so by not including a trailing slash.
The result of this is that the rule has blocked access to plugins whose slugs begin with search (i.e. /plugins/search-*/).

By adding a trailing slash here, it correctly applies the Disallow rule to /plugins/search/* instead.

Props digamberpradhan for noticing/reporting the issue.
See #5323.
Fixes #6413.

[dd32]

Noting I've refreshed googles cache of robots.txt and requested re-indexing for any matching plugins updated in the last 3 months (It's a manual process, so the other ~150 older plugins will just have to wait for the regularly scheduled google crawl - edit: although the older plugins have noindex in their robots tag anyway)