WordPress.org

Making WordPress.org

Opened 3 years ago

Closed 2 years ago

Last modified 12 months ago

#2330 closed defect (fixed)

Leverage the WordPress Cron / Cavalcade to run the plugin imports

Reported by: dd32 Owned by:
Milestone: Plugin Directory v3.0 Priority: normal
Component: Plugin Directory Keywords:
Cc:

Description

Currently the plugin directory is using a stand alone script which runs in a loop to process changes.

Historically this has caused problems during high-frequency-commits, so instead, lets switch the importer to run as a WordPress Cron Job, this will have the advantage that we can run many imports in parallel, it also brings complexity as a plugin shouldn't be imported twice at the same time.

On WordPress.org we're using Cavalcade to offload cron tasks from WordPress which has given jobs the ability to scale, it also means we can be sure that no two individual jobs are running in more than 1 location.

Change History (7)

#1 @dd32
3 years ago

In 4520:

Plugin Directory: Shift SVN Watching, Plugin Imports, and Plugin i18n imports to Jobs.

See #2330.

#2 @dd32
3 years ago

In 4521:

Plugin Directory: Fix a typo in the i18n importer [4520] by using a consistent variable name.

See #2330

#3 @dd32
3 years ago

In 4522:

Plugin Directory: Queue up plugin i18n import jobs after the main plugin import runs.

See #2330

#4 @dd32
3 years ago

In 4523:

Plugin Directory: Flush the Cavalcade jobs cache as these methods need to operate on fresh data from the database - currently they may be unaware of another job added while the current process was running.

See #2330.

#5 @dd32
2 years ago

  • Resolution set to fixed
  • Status changed from new to closed

This seems to be holding up pretty well.

#6 @obenland
2 years ago

In 4670:

Plugin Directory: Add missing constant PHP

Fixes a bug where imports can't be processed due to a missing path to php.

See #2330, [4520].

#7 @dd32
12 months ago

In 7370:

Plugin Directory: When re-queueing the next plugin import job, queue it for a minutes time rather than an hours time.

See https://wordpress.slack.com/archives/C02QB8GMM/p1530669226000099 for discussion.
See #2330.

Note: See TracTickets for help on using tickets.