#1578 closed task (blessed) (fixed)
Zip Files
Reported by: | obenland | Owned by: | |
---|---|---|---|
Milestone: | Priority: | high | |
Component: | Plugin Directory | Keywords: | |
Cc: |
Description
Generate zip files on the fly.
Change History (28)
#2
@
8 years ago
Remaining:
- Invalidate ZIP upon svn commit, if it touched that tag
- Some method for us to invalidate zips, could just be appending
?invalidate=true
.
#4
follow-up:
↓ 6
@
8 years ago
Another thing that needs to be done:
- Verify that the ZIPs generated ARE the same as the ones currently being generated, excluding file order differences.
The simplest way to achieve this would be to compare the filesize of *every* plugin trunk/tag and review any that are more than a few bytes difference.
#6
in reply to:
↑ 4
@
8 years ago
Replying to dd32:
Another thing that needs to be done:
- Verify that the ZIPs generated ARE the same as the ones currently being generated, excluding file order differences.
The simplest way to achieve this would be to compare the filesize of *every* plugin trunk/tag and review any that are more than a few bytes difference.
I've started a script to verify that the filesize of the compressed files within the zip files of the latest version of every plugin is the same.
The MD5's of the zip files differ, as does the size of the ZIPs as the new ones don't have a zip comment and have their files ordered in a consistent manner (which differs from existing).
#7
follow-up:
↓ 15
@
8 years ago
Fixed in r3291
So far one issue that's come up is invalid filenames, for example:
$ svn export https://plugins.svn.wordpress.org/1beyt/trunk/ . --force A . A inc A inc/Poems.sql A inc/widget.php A inc/ob_shortcode.php A 1beyt.php A icon-256x256.png A readme.txt A help.php A screenshot-1.png A uninstall.php A images A images/icon.png A images/logo.png svn: E000022: Can't convert string from 'UTF-8' to native encoding: svn: E000022: banner-772?\195?\151250.png
For some reason, the new scripts hit that and throw an error, whereas the existing zip generator sails through and adds that to the ZIP. I can't see why, but I'm guessing one of the CLI LC
environmental variables is set to something..
(Sidenote: Yes, the filename is not what's expected in the plugin directory, and it's even in the wrong location, but that's pretty common in SVN at present, I've even seen it on plugins from well known developers.)
#8
@
8 years ago
Fixed in r3213
Another bug, the stable_tag
can be set to all sorts of amazing things, including this scenario:
- https://plugins.svn.wordpress.org/bulletproof-security/trunk/readme.txt =>
Stable tag: .53.5
will result in a ZIP filename ofbulletproof-security.0.53.5.zip
note the additional prefixed 0 which is neither in the readme nor in the SVN tag.
Seems like the fix here is to:
- Prefix the 0 when the
stable_tag
starts with.
- When generating the zip, if specified tag (
/tags/0.53.5
) doesn't exist and is0.
prefixed, check to see if the tag without the zero exists (/tags/.53.5
).
#9
@
8 years ago
Many of the existing ZIPs include tmp-$slug-readme.pot
incorrectly, which was a bug late last year where the i18n imports didn't clean up after themselves before the ZIPs were created.
I'm also running into a lot of consistency issues where ZIPs are missing from 1 of the 5 download nodes.
both of those are making verifying that everything is sane a bit harder.
#15
in reply to:
↑ 7
@
8 years ago
Replying to dd32:
So far one issue that's come up is invalid filenames, for example:
...
For some reason, the new scripts hit that and throw an error, whereas the existing zip generator sails through and adds that to the ZIP. I can't see why, but I'm guessing one of the CLILC
environmental variables is set to something..
Indeed, the existing plugin directory update handlers have this in them:
export LC_CTYPE="en_US.UTF-8" export LANG="en_US.UTF-8"
[3214] allowed the SVN command to complete, but the resulting ZIP still had invalid filename issues, as it wasn't being set for the ZIP commands.
The above LANG isn't available on my test environments, so i'll come back to that..
This ticket was mentioned in Slack in #meta by obenland. View the logs.
8 years ago
This ticket was mentioned in Slack in #meta by dd32. View the logs.
8 years ago
#20
@
8 years ago
All the known bugs have been fixed here.
I'll need to update the comparison script to attempt to ignore the tmp-*.pot
files and run it again to see if it can find any other new amazing issues.
This ticket was mentioned in Slack in #meta by obenland. View the logs.
8 years ago
#24
@
8 years ago
- Resolution set to fixed
- Status changed from new to closed
Will re-open or create new tickets for any further issues.
#25
@
8 years ago
A decision was made that for launch, we'd like to remove the ZIP rebuild-on-demand and instead store creates ZIPs in a centralised SVN repository.
This allows better caching, as the ZIP can be stored on load balancers caches rather than on each individual web server and increases reliability as it's harder for one server to contain a ZIP which doesn't match all the others (which has happened in the past).
Additionally, although the ZIPs were being created in a way that their checksums remained consistent, by only building it once and storing it, it'll protect against future scenario's where the checksums no longer match between servers (although tag commits and trunk alterations obviously change the zip checksums).
In 3081: