Block crawlers from tags and other paged results

[bpetty]

Specifically in regards to the theme and plugin directories, it would be helpful if the paginated results for plugins and themes weren't the results returned from search engines (rather, the plugin and theme pages themselves).

This is mostly just a problem with tag pages from what I can tell, and those can certainly be removed without making any plugins or themes unreachable from search crawlers:

Disallow: /themes/tags
Disallow: /plugins/tags

Those paginated tag results often come up in search results since obviously, if you're searching with keywords based on those tags, the tag pages themselves contain those keywords in the title and header, while the theme or plugin page itself doesn't.

I would also suggest the popular and newest paginated results, but doing these as well might make it impossible for crawlers to reach and index many plugin and theme pages...

Disallow: /themes/browse/popular/page
Disallow: /themes/browse/new/page
Disallow: /plugins/browse/popular/page
Disallow: /plugins/browse/new/page

I would like to at least see tags disallowed though.

[iandunn]

Can you post some example queries that show the problem? I'm not seeing any right off the bat.

[bpetty]

Example search

[bpetty]

There's another example taken from this Torque article, so I know I'm not the only one seeing it: ​http://torquemag.io/write-irresistible-plugin-title-downloads-guaranteed/

[joostdevalk]

I honestly don't see why we'd block those, we might need to _improve_ on them, but my proposal would be to keep them wide open.

[ocean90]

Per Joost comment.