Limit pagination for plugin search results

[ocean90]

Example: ​https://wordpress.org/plugins/search/help/page/643/ shows results but ​https://wordpress.org/plugins/search/help/page/644/ doesn't. It redirects to ​https://wordpress.org/plugins/search/.

Based on the error message there's a limit in Elasticsearch that prevents an offset (from parameter) above 9000.

{"error":"Bad Request","message":"Elasticsearch cannot page past the first 9000 results."}

On page 643 it's 8359 ((643-1)*14) and 644 has 9002 ((644-1)*14). Forcing it to 9000 still returns results.

[obenland]

Related on the topic of paginating past 100 pages: #3414

[ocean90]

@gibrown Can you confirm that the limit has been reduced to 9k?

[gibrown]

Ya that is correct. The exact limits are here: ​https://developer.wordpress.com/docs/elasticsearch/elasticsearch-queries/ For all intents and purposes wp.org is treated as another VIP (though it is kinda a special case).

Additional background:

• this limit was always there (ES has a default limit of 10k), but we were just failing the query rather than returning an error message that really said you were hitting a limit.
• We decided to set a limit of 9k for from and 1k for size rather than the more complicated (to explain and check) limit on each separately
• Latest Jetpack (5.8 just released) issues a 404 when this condition is hit. We are running an old fork of that code, but updating to the real code would be another good way to fix this.

That patch looks good to me for an immediate fix.

[ocean90]

In 6590:

Plugins Directory: Adjust pagination handling for search results to match updated Elasticsearch limits.

See ​https://developer.wordpress.com/docs/elasticsearch/elasticsearch-queries/.
Fixes #3442.