Making WordPress.org

Opened 4 months ago

Last modified 7 weeks ago

#5458 new defect

Redirects should not trigger 429 responses, and rate limiting should be relaxed.

Reported by: jonoaldersonwp Owned by:
Milestone: Priority: highest omg bbq
Component: General Keywords: seo

Description (last modified by jonoaldersonwp)

Requests to https://wordpress.org/tags/privacy/page/15 are redirected to https://wordpress.org/support/topic-tag/privacy/page/15/.

However, in the process, there are several interstitial redirects (the change in path, and, appending the trailing slash).

This frequently triggers a 429 error for Google (and presumably other systems).

To address this:

  • Rate limiting on 301 redirects should be removed (or radically relaxed).
  • System-level redirect rules (altering paths, appending trailing slashes, case normalisation) should be consolidated into a single redirect.

This also appears to apply to any paginated request (e.g., https://wordpress.org/support/topic/my-newest-blog/page/2/); which makes this even more damaging, significant, and urgent to fix.

Given that we 'removed' paginated states beyond 50, the performance implications of this should be somewhat mitigated. If there are still performance concerns, there's no shortage of opportunities to optimise other parts of the site(s)/stack to recoup our losses.

Change History (9)

#1 @jonoaldersonwp
4 months ago

NB, I suspect this is responsible for hundreds of small oddities that I've been seeing but unable to diagnose over the past few months. Aside from fixing the underlying issue, this would clean up our reporting significantly (and thus make it easier to find/manage other issues).

#2 @jonoaldersonwp
3 months ago

As we gradually fix other areas, I'm seeing more evidence emerging that this is hurting crawling and discoverability. This remains a (very) high priority.

#3 @jonoaldersonwp
3 months ago

  • Description modified (diff)

This ticket was mentioned in Slack in #meta by jonoaldersonwp. View the logs.

3 months ago

#5 @jonoaldersonwp
3 months ago

  • Description modified (diff)
  • Summary changed from Redirects should not trigger 429 responses to Redirects should not trigger 429 responses, and rate limiting should be relaxed.

#6 follow-up: @dd32
3 months ago

Just noting that it's impossible to rate limit redirects at a different rate than non-redirects.

Due to how nginx rate limiting works, the counters are updating prior to the request being made, and the blocking is applied prior to the system processing the request. As a result, the module does not know if the request is a redirect either at the point of bumping the request counters or blocking the request from happening.

The only way to resolve that is to increase the allowed rate limits, or fix the clients which are requesting pages at a too high rate that ultimately triggers the blocking.

Consolidating redirects / etc can somewhat be done (and has been), but the vast majority are not worth the effort or are almost impossible to do so due to the varied places that the redirects happen, as has been explained many times.

#7 in reply to: ↑ 6 @dd32
3 months ago

Replying to dd32:

Due to how nginx rate limiting works...

Looks like I might've been wrong, server-level redirects probably bypass the rate limiting already as per https://trac.nginx.org/nginx/ticket/1834 so any redirect requests that are being rate limited here, are redirects from PHP which means many of the redirects are already consolidated or unaffected by the rate limiting.

#8 @jonoaldersonwp
3 months ago

So, how do we unpick this?
Feels like the easiest fix would just be to (significantly) relax the rate-limiting for Google?

#9 @jonoaldersonwp
7 weeks ago

  • Priority changed from high to highest omg bbq

This is resulting in large numbers of errors in Google Search Console, and continues to contribute to the serious delays we see in discovery, crawling, indexing and consolidation.

Note: See TracTickets for help on using tickets.