Zip Files

[obenland]

Generate zip files on the fly.

[dd32]

In 3081:

Plugin Directory: Add an initial set of scripts to build & serve ZIP files.

See #505 - The new directory doesn't build ZIPs for branches.
See #1578

[dd32]

Remaining:

• Invalidate ZIP upon svn commit, if it touched that tag
• Some method for us to invalidate zips, could just be appending ?invalidate=true.

[dd32]

Another thing that needs to be done:

• Verify that the ZIPs generated ARE the same as the ones currently being generated, excluding file order differences.

The simplest way to achieve this would be to compare the filesize of *every* plugin trunk/tag and review any that are more than a few bytes difference.

[dd32]

Replying to dd32:

Another thing that needs to be done:

• Verify that the ZIPs generated ARE the same as the ones currently being generated, excluding file order differences.

The simplest way to achieve this would be to compare the filesize of *every* plugin trunk/tag and review any that are more than a few bytes difference.

I've started a script to verify that the filesize of the compressed files within the zip files of the latest version of every plugin is the same.
The MD5's of the zip files differ, as does the size of the ZIPs as the new ones don't have a zip comment and have their files ordered in a consistent manner (which differs from existing).

[dd32]

Fixed in r3291

---

So far one issue that's come up is invalid filenames, for example:

$ svn export https://plugins.svn.wordpress.org/1beyt/trunk/ . --force
A    .
A    inc
A    inc/Poems.sql
A    inc/widget.php
A    inc/ob_shortcode.php
A    1beyt.php
A    icon-256x256.png
A    readme.txt
A    help.php
A    screenshot-1.png
A    uninstall.php
A    images
A    images/icon.png
A    images/logo.png
svn: E000022: Can't convert string from 'UTF-8' to native encoding:
svn: E000022: banner-772?\195?\151250.png

For some reason, the new scripts hit that and throw an error, whereas the existing zip generator sails through and adds that to the ZIP. I can't see why, but I'm guessing one of the CLI LC environmental variables is set to something..

(Sidenote: Yes, the filename is not what's expected in the plugin directory, and it's even in the wrong location, but that's pretty common in SVN at present, I've even seen it on plugins from well known developers.)

[dd32]

Fixed in r3212

---

Another bug, the stable_tag can be set to all sorts of amazing things, including this scenario:

• ​https://plugins.svn.wordpress.org/bulletproof-security/trunk/readme.txt => Stable tag: .53.5 will result in a ZIP filename of bulletproof-security.0.53.5.zip note the additional prefixed 0 which is neither in the readme nor in the SVN tag.

Seems like the fix here is to:

• Prefix the 0 when the stable_tag starts with .
• When generating the zip, if specified tag (/tags/0.53.5) doesn't exist and is 0. prefixed, check to see if the tag without the zero exists (/tags/.53.5).

[dd32]

Many of the existing ZIPs include tmp-$slug-readme.pot incorrectly, which was a bug late last year where the i18n imports didn't clean up after themselves before the ZIPs were created.

I'm also running into a lot of consistency issues where ZIPs are missing from 1 of the 5 download nodes.

both of those are making verifying that everything is sane a bit harder.

[dd32]

In 3189:

Plugin Directory: ZIPs: When Building ZIPs do not include any svn:externals.

See #1578

[dd32]

In 3210:

Plugin Directory: ZIPs: Budypress is allowed to have svn:externals as it's a sister project.

See #1578

[dd32]

In 3212:

Plugin Directory: ZIPs: Handle stable_tag's which are set to things like .12.3 rather than the expected 0.12.3.

See #1578

[dd32]

In 3214:

Plugin Directory: When executing SVN functions, execute them with a UTF-8 language locale set, this allows for plugins with non-ascii filenames to be operated upon.

See #1578

[dd32]

In 3215:

Plugins Directory: Revert [3214] as although it works, at least one one host, it doesn't fully solve the issue.

See #1578

[dd32]

Replying to dd32:

So far one issue that's come up is invalid filenames, for example:
...
For some reason, the new scripts hit that and throw an error, whereas the existing zip generator sails through and adds that to the ZIP. I can't see why, but I'm guessing one of the CLI LC environmental variables is set to something..

Indeed, the existing plugin directory update handlers have this in them:

export LC_CTYPE="en_US.UTF-8"
export LANG="en_US.UTF-8"

[3214] allowed the SVN command to complete, but the resulting ZIP still had invalid filename issues, as it wasn't being set for the ZIP commands.
The above LANG isn't available on my test environments, so i'll come back to that..

[dd32]

In 3291:

Plugin Directory: Execute the ZIP & SVN commands with an appropriate UTF-8 character set.

See #1578

[dd32]

All the known bugs have been fixed here.

I'll need to update the comparison script to attempt to ignore the tmp-*.pot files and run it again to see if it can find any other new amazing issues.

[dd32]

In 3321:

Plugin Directory: When building ZIP files, use a custom implementation of tempnam() to accomodate plugins which have longer filenames.
Previously a plugin with a slug and version combination exceeding 60 characters would cause a build failure.
See #1578

[dd32]

In 3367:

Plugin Directory: Add an internal API for managing ZIP files on each web node.

See #1578, #1720

[dd32]

Will re-open or create new tickets for any further issues.

[dd32]

A decision was made that for launch, we'd like to remove the ZIP rebuild-on-demand and instead store creates ZIPs in a centralised SVN repository.

This allows better caching, as the ZIP can be stored on load balancers caches rather than on each individual web server and increases reliability as it's harder for one server to contain a ZIP which doesn't match all the others (which has happened in the past).

Additionally, although the ZIPs were being created in a way that their checksums remained consistent, by only building it once and storing it, it'll protect against future scenario's where the checksums no longer match between servers (although tag commits and trunk alterations obviously change the zip checksums).

[dd32]

In 5147:

Plugin Directory: ZIPs: Do not build zips on demand, instead store them within a SVN repository.

See #1578

[samuelsidler]

Milestone Plugin Directory v3 - M5 deleted

[dd32]

In 5246:

Plugin Directory: When a version of '.1' is passed to the ZIP builder, correct it to '0.1'.
Although the rest of the directory handles these version numbers correctly, if the ZIP builder is passed the '.1' variant it created the incorrect ZIP filenames.
This didn't affect the plugin directory, but did occur when I bulk-built ZIP files from a svn tag listing, causing some plugins to have incorrect ZIP filenames generated (resulting in 404's).

See #1578