Add Content-MD5 headers to downloads

[nacin]

Ths was not easy to do in nginx. x-accel-redirect drops the content-md5 header. See ​#WP20074.

[Otto42]

I know little to nothing about how our particular nginx config is setup, but:

​https://stackoverflow.com/questions/24507157/adding-custom-http-headers-to-nginx-x-accel-redirect

add_header Content-MD5 $upstream_http_content_md5;

[dd32]

Replying to Otto42:

I know little to nothing about how our particular nginx config is setup, but:

​https://stackoverflow.com/questions/24507157/adding-custom-http-headers-to-nginx-x-accel-redirect

add_header Content-MD5 $upstream_http_content_md5;

That does what's needed in our setup, Once upon a time it required some pretty nasty hacky location blocks (basically double proxying the x-accel-redirect) in order to work, but not no more.

IIRC we didn't enable this on downloads.wordpress.org due to concern over the error message in core when the checksum would fail.

[georgestephanis]

I seem to recall something around April or so when mdawaffe or nacin remarked that plugin md5s from different servers didn't match -- due to something marking either a compilation timestamp or marking which server built the zip -- I don't recall exactly, but there was a reason that the same plugin downloaded from different places didn't have matching md5s.

Which just means you wouldn't be able to compare the file md5 & header md5 with an external hash listing, they should still match up but that's all coming from the same request.

I think.

[Otto42]

Indeed, because each server gets the files from SVN and then builds their own ZIP files, the timestamps on the individual files in the ZIP will be different, even if the content is the same. Thus, the ZIPs themselves will be different.

To do this for all plugins/themes, we'd need to only build the ZIPs once and store them in a location all the servers can access. Or dedicate downloads to a single server or set of them.

[dd32]

I seem to recall something around April or so when mdawaffe or nacin remarked that plugin md5s from different servers didn't match

Yep, but I don't think this is a problem, It represents the MD5 of the content of the request, not a canonical MD5 of the package.

[bgermann]

To address the different ZIP timestamps one could also set the SVN use-commit-times option on those machines.

[Otto42]

That wouldn't work in our case, because while svn --config-option config:miscellany:use-commit-times=yes export http://whatever does indeed preserve the commit times for the individual files, plugins and themes in the ZIPs are put into a directory (named with the plugin slug) which is created at the time of the ZIP. That directory will still have the current time for that particular server.

Note that this would not be an issue with core downloads, those ZIPs are built at release time and manually committed, they're not built by the system on-the-fly.

[bgermann]

Are the build scripts publicly available? I search for them but did not find any. I would love to have a uniform download and would supply a patch for it.

[Otto42]

Not for plugins and themes yet, no.

[bgermann]

You could get the last commit date of the branch/tag which you are generating the zip file for and set the directory's date with touch -d or touch -t.

[Otto42]

#1410 was marked as a duplicate.

[ocean90]

In [dotorg11195]:

Generate consitent plugin/theme ZIP files regardless of which server or time the zip is generated at.
This change causes all zip files to have the same hash, as long as they're generated from the same SVN revision.

@dd32, @Otto42: Is this something we can implement now?

[dd32]

Yes, we should be able to implement this now thanks to [dotorg11195]. I hesitated after that commit, as I wasn't fully sure if it would have the intended effect, we should do a survey of the zips to check they all match and then go for it.

[ocean90]

Note: [dotorg11195] applies only to new builds, plugins with no recent build still can have different packages.

[dd32]

As the MD5 won't be generated on-the-fly and only at package generation time, it'll only be for packages built after when it's enabled.

As part of the process I'll look into having the zips rebuilt where needed.

[dd32]

To summarize, when creating a zip file on-the-fly, there's several things you need to do

1. svn export will set files to their last-commit-date, but directories to NOW(). Update directories timestamps to a known date (I chose the timestamp of the last commit to that theme/plugin).
2. When copying files between directories, use --preserve=timestamps to preserve the timestamps rather than having them set to NOW().
3. Using zip specify the -X parameter to strip filesystem-specific things such as user/group's and access time from the zip, as these are not needed and may vary.
4. When creating the zip file, zip will add the files in the order it finds them on disk, however even if you specify a file order via STDIN, it'll add them in a seemingly random (and system dependant) order. Files need to be sorted and added individually to ensure that it's consistent.

The theme zip files appear to be being generated consistently now, plugin zip generation needs #4 applied to it still.

[bgermann]

Is this the right place for modifications? https://meta.trac.wordpress.org/browser/sites/trunk/wordpress.org/public_html/wp-content/plugins/plugin-directory/zip/class-builder.php

[Otto42]

This should definitely be doable now with the changes made to the ZIP building process.

I believe the original nginx rules mentioned should still work. Do we need to ping systems about it?

[dd32]

With the more recent changes made to downloads, this is no longer as easy as it previously was.

ZIPs are now reverse-proxied to a SVN server, and the server responding to the request no longer has access to the file to generate the MD5.
We'd probably have to cache ZIP md5's in the middle somewhere.

Personally I'm -1 on implementing anything here now, and would instead prefer to see the next step after Content-MD5 being implemented, a signed hash of the zip instead.

[dd32]

Closing as wontfix, as per the above comment it's no where as easy as it was in the past to add.

We'll be supporting signatures for plugins soon which will cover this with a better checksum.

[Otto42]

#4662 was marked as a duplicate.