Opened 4 months ago
Last modified 7 weeks ago
#5458 new defect
Redirects should not trigger 429 responses, and rate limiting should be relaxed.
Reported by: |
|
Owned by: | |
---|---|---|---|
Milestone: | Priority: | highest omg bbq | |
Component: | General | Keywords: | seo |
Cc: |
Description (last modified by )
Requests to https://wordpress.org/tags/privacy/page/15 are redirected to https://wordpress.org/support/topic-tag/privacy/page/15/.
However, in the process, there are several interstitial redirects (the change in path, and, appending the trailing slash).
This frequently triggers a 429 error for Google (and presumably other systems).
To address this:
- Rate limiting on 301 redirects should be removed (or radically relaxed).
- System-level redirect rules (altering paths, appending trailing slashes, case normalisation) should be consolidated into a single redirect.
This also appears to apply to any paginated request (e.g., https://wordpress.org/support/topic/my-newest-blog/page/2/); which makes this even more damaging, significant, and urgent to fix.
Given that we 'removed' paginated states beyond 50, the performance implications of this should be somewhat mitigated. If there are still performance concerns, there's no shortage of opportunities to optimise other parts of the site(s)/stack to recoup our losses.
Change History (9)
#2
@
3 months ago
As we gradually fix other areas, I'm seeing more evidence emerging that this is hurting crawling and discoverability. This remains a (very) high priority.
This ticket was mentioned in Slack in #meta by jonoaldersonwp. View the logs.
3 months ago
#5
@
3 months ago
- Description modified (diff)
- Summary changed from Redirects should not trigger 429 responses to Redirects should not trigger 429 responses, and rate limiting should be relaxed.
#6
follow-up:
↓ 7
@
3 months ago
Just noting that it's impossible to rate limit redirects at a different rate than non-redirects.
Due to how nginx rate limiting works, the counters are updating prior to the request being made, and the blocking is applied prior to the system processing the request. As a result, the module does not know if the request is a redirect either at the point of bumping the request counters or blocking the request from happening.
The only way to resolve that is to increase the allowed rate limits, or fix the clients which are requesting pages at a too high rate that ultimately triggers the blocking.
Consolidating redirects / etc can somewhat be done (and has been), but the vast majority are not worth the effort or are almost impossible to do so due to the varied places that the redirects happen, as has been explained many times.
#7
in reply to:
↑ 6
@
3 months ago
Replying to dd32:
Due to how nginx rate limiting works...
Looks like I might've been wrong, server-level redirects probably bypass the rate limiting already as per https://trac.nginx.org/nginx/ticket/1834 so any redirect requests that are being rate limited here, are redirects from PHP which means many of the redirects are already consolidated or unaffected by the rate limiting.
NB, I suspect this is responsible for hundreds of small oddities that I've been seeing but unable to diagnose over the past few months. Aside from fixing the underlying issue, this would clean up our reporting significantly (and thus make it easier to find/manage other issues).