Opened 10 years ago

Closed 8 years ago

#398 closed enhancement (wontfix)

Block crawlers from tags and other paged results

Reported by: bpetty's profile bpetty Owned by:
Milestone: Priority: normal
Component: General Keywords: close


Specifically in regards to the theme and plugin directories, it would be helpful if the paginated results for plugins and themes weren't the results returned from search engines (rather, the plugin and theme pages themselves).

This is mostly just a problem with tag pages from what I can tell, and those can certainly be removed without making any plugins or themes unreachable from search crawlers:

Disallow: /themes/tags
Disallow: /plugins/tags

Those paginated tag results often come up in search results since obviously, if you're searching with keywords based on those tags, the tag pages themselves contain those keywords in the title and header, while the theme or plugin page itself doesn't.

I would also suggest the popular and newest paginated results, but doing these as well might make it impossible for crawlers to reach and index many plugin and theme pages...

Disallow: /themes/browse/popular/page
Disallow: /themes/browse/new/page
Disallow: /plugins/browse/popular/page
Disallow: /plugins/browse/new/page

I would like to at least see tags disallowed though.

Attachments (2)

yellow-theme-google-search.png (57.0 KB) - added by bpetty 10 years ago.
Example search
wporg-tags-from-plugins.png (294.2 KB) - added by bpetty 10 years ago.

Download all attachments as: .zip

Change History (9)

This ticket was mentioned in IRC in #wordpress-meta by sams. View the logs.

10 years ago

#2 @iandunn
10 years ago

Can you post some example queries that show the problem? I'm not seeing any right off the bat.

10 years ago

Example search

#3 @bpetty
10 years ago

There's another example taken from this Torque article, so I know I'm not the only one seeing it:

#4 @joostdevalk
9 years ago

  • Keywords close added

I honestly don't see why we'd block those, we might need to _improve_ on them, but my proposal would be to keep them wide open.

This ticket was mentioned in Slack in #meta by joostdevalk. View the logs.

9 years ago

This ticket was mentioned in Slack in #meta by ocean90. View the logs.

8 years ago

#7 @ocean90
8 years ago

  • Resolution set to wontfix
  • Status changed from new to closed

Per Joost comment.

Note: See TracTickets for help on using tickets.