Opened 8 years ago

Closed 7 years ago

Last modified 7 years ago

#1720 closed task (blessed) (fixed)

post-commit svn hook

Reported by: dd32's profile dd32 Owned by:
Milestone: Priority: highest omg bbq
Component: Plugin Directory Keywords:


After each commit we'll need to reparse the plugins affected by the changeset.

The existing plugins directory doesn't use a post-commit hook, rather, it simply parses from last-parsed-revision to current-revision.
It does this as parsing is an expensive process and processing each plugin as the commit happens will result in a large number of processes running in parallel during high commit scenario's.

With that in mind, we need to

  • Upon SVN commit, mark a plugin as needing to be parsed
  • Process the i18n imports
  • Invalidate ZIPs affected by the commit
  • Not run the import on svn commit, but rather through some form of queue system (a job system such as cavalcade preferably or a custom cron job if need be)
  • Not run too many imports concurrently to avoid resource exhaustion, or plugins.svn becoming unresponsive (it happens)
  • Not run two imports for the same plugin at the same time (ie. commit 123 and commit 1234 both touch plugin-x, shouldn't result in two parallel imports, but one after the other *at most*)

Not strictly part of this, but this is also where we should kick off any code-scanning jobs.

Change History (10)

This ticket was mentioned in Slack in #meta by pento. View the logs.

8 years ago

This ticket was mentioned in Slack in #meta by obenland. View the logs.

8 years ago

#3 @obenland
8 years ago

  • Milestone set to Plugin Directory v3 - M5

This ticket was mentioned in Slack in #meta by dd32. View the logs.

7 years ago

#5 @dd32
7 years ago

In 3367:

Plugin Directory: Add an internal API for managing ZIP files on each web node.

See #1578, #1720

#6 @dd32
7 years ago

In 3370:

Plugin Directory: Add a SVN method to fetch the log of a range of revisions.

See #1720

#7 @dd32
7 years ago

In 3373:

Plugin Directory: Add a SVN watching script to process & import plugins to the plugin directory after each commit.

This doesn't act as a post-commit hook, but instead is designed to be run in an endless loop.

See #1720

#8 @dd32
7 years ago

  • Resolution set to fixed
  • Status changed from new to closed

#9 @dd32
7 years ago

In 4195:

Plugin Directory: Fix a typo in the SVN watcher.

See #1720

#10 @samuelsidler
7 years ago

  • Milestone Plugin Directory v3 - M5 deleted

Milestone Plugin Directory v3 - M5 deleted

Note: See TracTickets for help on using tickets.