Making WordPress.org

Opened 9 years ago

Closed 8 years ago

Last modified 8 years ago

#1578 closed task (blessed) (fixed)

Zip Files

Reported by: obenland's profile obenland Owned by:
Milestone: Priority: high
Component: Plugin Directory Keywords:
Cc:

Description

Generate zip files on the fly.

Change History (28)

#1 @dd32
9 years ago

In 3081:

Plugin Directory: Add an initial set of scripts to build & serve ZIP files.

See #505 - The new directory doesn't build ZIPs for branches.
See #1578

#2 @dd32
8 years ago

Remaining:

  • Invalidate ZIP upon svn commit, if it touched that tag
  • Some method for us to invalidate zips, could just be appending ?invalidate=true.

#3 @dd32
8 years ago

  • Type changed from enhancement to task

#4 follow-up: @dd32
8 years ago

Another thing that needs to be done:

  • Verify that the ZIPs generated ARE the same as the ones currently being generated, excluding file order differences.

The simplest way to achieve this would be to compare the filesize of *every* plugin trunk/tag and review any that are more than a few bytes difference.

#5 @dd32
8 years ago

  • Priority changed from normal to high

#6 in reply to: ↑ 4 @dd32
8 years ago

Replying to dd32:

Another thing that needs to be done:

  • Verify that the ZIPs generated ARE the same as the ones currently being generated, excluding file order differences.

The simplest way to achieve this would be to compare the filesize of *every* plugin trunk/tag and review any that are more than a few bytes difference.

I've started a script to verify that the filesize of the compressed files within the zip files of the latest version of every plugin is the same.
The MD5's of the zip files differ, as does the size of the ZIPs as the new ones don't have a zip comment and have their files ordered in a consistent manner (which differs from existing).

#7 follow-up: @dd32
8 years ago

Fixed in r3291


So far one issue that's come up is invalid filenames, for example:

$ svn export https://plugins.svn.wordpress.org/1beyt/trunk/ . --force
A    .
A    inc
A    inc/Poems.sql
A    inc/widget.php
A    inc/ob_shortcode.php
A    1beyt.php
A    icon-256x256.png
A    readme.txt
A    help.php
A    screenshot-1.png
A    uninstall.php
A    images
A    images/icon.png
A    images/logo.png
svn: E000022: Can't convert string from 'UTF-8' to native encoding:
svn: E000022: banner-772?\195?\151250.png

For some reason, the new scripts hit that and throw an error, whereas the existing zip generator sails through and adds that to the ZIP. I can't see why, but I'm guessing one of the CLI LC environmental variables is set to something..

(Sidenote: Yes, the filename is not what's expected in the plugin directory, and it's even in the wrong location, but that's pretty common in SVN at present, I've even seen it on plugins from well known developers.)

Last edited 8 years ago by dd32 (previous) (diff)

#8 @dd32
8 years ago

Fixed in r3213


Another bug, the stable_tag can be set to all sorts of amazing things, including this scenario:

Seems like the fix here is to:

  • Prefix the 0 when the stable_tag starts with .
  • When generating the zip, if specified tag (/tags/0.53.5) doesn't exist and is 0. prefixed, check to see if the tag without the zero exists (/tags/.53.5).
Version 3, edited 8 years ago by dd32 (previous) (next) (diff)

#9 @dd32
8 years ago

Many of the existing ZIPs include tmp-$slug-readme.pot incorrectly, which was a bug late last year where the i18n imports didn't clean up after themselves before the ZIPs were created.

I'm also running into a lot of consistency issues where ZIPs are missing from 1 of the 5 download nodes.

both of those are making verifying that everything is sane a bit harder.

#10 @dd32
8 years ago

In 3189:

Plugin Directory: ZIPs: When Building ZIPs do not include any svn:externals.

See #1578

#11 @dd32
8 years ago

In 3210:

Plugin Directory: ZIPs: Budypress is allowed to have svn:externals as it's a sister project.

See #1578

#12 @dd32
8 years ago

In 3212:

Plugin Directory: ZIPs: Handle stable_tag's which are set to things like .12.3 rather than the expected 0.12.3.

See #1578

#13 @dd32
8 years ago

In 3214:

Plugin Directory: When executing SVN functions, execute them with a UTF-8 language locale set, this allows for plugins with non-ascii filenames to be operated upon.

See #1578

#14 @dd32
8 years ago

In 3215:

Plugins Directory: Revert [3214] as although it works, at least one one host, it doesn't fully solve the issue.

See #1578

#15 in reply to: ↑ 7 @dd32
8 years ago

Replying to dd32:

So far one issue that's come up is invalid filenames, for example:
...
For some reason, the new scripts hit that and throw an error, whereas the existing zip generator sails through and adds that to the ZIP. I can't see why, but I'm guessing one of the CLI LC environmental variables is set to something..

Indeed, the existing plugin directory update handlers have this in them:

export LC_CTYPE="en_US.UTF-8"
export LANG="en_US.UTF-8"

[3214] allowed the SVN command to complete, but the resulting ZIP still had invalid filename issues, as it wasn't being set for the ZIP commands.
The above LANG isn't available on my test environments, so i'll come back to that..

This ticket was mentioned in Slack in #meta by obenland. View the logs.


8 years ago

#17 @obenland
8 years ago

  • Milestone set to Plugin Directory v3 - M5

#18 @dd32
8 years ago

In 3291:

Plugin Directory: Execute the ZIP & SVN commands with an appropriate UTF-8 character set.

See #1578

This ticket was mentioned in Slack in #meta by dd32. View the logs.


8 years ago

#20 @dd32
8 years ago

All the known bugs have been fixed here.

I'll need to update the comparison script to attempt to ignore the tmp-*.pot files and run it again to see if it can find any other new amazing issues.

#21 @dd32
8 years ago

In 3321:

Plugin Directory: When building ZIP files, use a custom implementation of tempnam() to accomodate plugins which have longer filenames.
Previously a plugin with a slug and version combination exceeding 60 characters would cause a build failure.
See #1578

#22 @dd32
8 years ago

In 3367:

Plugin Directory: Add an internal API for managing ZIP files on each web node.

See #1578, #1720

This ticket was mentioned in Slack in #meta by obenland. View the logs.


8 years ago

#24 @dd32
8 years ago

  • Resolution set to fixed
  • Status changed from new to closed

Will re-open or create new tickets for any further issues.

#25 @dd32
8 years ago

A decision was made that for launch, we'd like to remove the ZIP rebuild-on-demand and instead store creates ZIPs in a centralised SVN repository.

This allows better caching, as the ZIP can be stored on load balancers caches rather than on each individual web server and increases reliability as it's harder for one server to contain a ZIP which doesn't match all the others (which has happened in the past).

Additionally, although the ZIPs were being created in a way that their checksums remained consistent, by only building it once and storing it, it'll protect against future scenario's where the checksums no longer match between servers (although tag commits and trunk alterations obviously change the zip checksums).

#26 @dd32
8 years ago

In 5147:

Plugin Directory: ZIPs: Do not build zips on demand, instead store them within a SVN repository.

See #1578

#27 @samuelsidler
8 years ago

  • Milestone Plugin Directory v3 - M5 deleted

Milestone Plugin Directory v3 - M5 deleted

#28 @dd32
8 years ago

In 5246:

Plugin Directory: When a version of '.1' is passed to the ZIP builder, correct it to '0.1'.
Although the rest of the directory handles these version numbers correctly, if the ZIP builder is passed the '.1' variant it created the incorrect ZIP filenames.
This didn't affect the plugin directory, but did occur when I bulk-built ZIP files from a svn tag listing, causing some plugins to have incorrect ZIP filenames generated (resulting in 404's).

See #1578

Note: See TracTickets for help on using tickets.