WordPress.org

Making WordPress.org

Opened 3 years ago

Last modified 11 days ago

#619 new enhancement

Add Content-MD5 headers to downloads

Reported by: nacin Owned by:
Milestone: Plugin Directory v3 - Future Priority: normal
Component: Plugin Directory Keywords: 2nd-opinion
Cc:

Description

Ths was not easy to do in nginx. x-accel-redirect drops the content-md5 header. See #WP20074.

Change History (21)

#1 follow-up: @Otto42
3 years ago

I know little to nothing about how our particular nginx config is setup, but:

https://stackoverflow.com/questions/24507157/adding-custom-http-headers-to-nginx-x-accel-redirect

add_header Content-MD5 $upstream_http_content_md5;

#2 in reply to: ↑ 1 @dd32
3 years ago

Replying to Otto42:

I know little to nothing about how our particular nginx config is setup, but:

https://stackoverflow.com/questions/24507157/adding-custom-http-headers-to-nginx-x-accel-redirect

add_header Content-MD5 $upstream_http_content_md5;

That does what's needed in our setup, Once upon a time it required some pretty nasty hacky location blocks (basically double proxying the x-accel-redirect) in order to work, but not no more.

IIRC we didn't enable this on downloads.wordpress.org due to concern over the error message in core when the checksum would fail.

#3 @georgestephanis
3 years ago

I seem to recall something around April or so when mdawaffe or nacin remarked that plugin md5s from different servers didn't match -- due to something marking either a compilation timestamp or marking which server built the zip -- I don't recall exactly, but there was a reason that the same plugin downloaded from different places didn't have matching md5s.

Which just means you wouldn't be able to compare the file md5 & header md5 with an external hash listing, they should still match up but that's all coming from the same request.

I think.

#4 @Otto42
3 years ago

Indeed, because each server gets the files from SVN and then builds their own ZIP files, the timestamps on the individual files in the ZIP will be different, even if the content is the same. Thus, the ZIPs themselves will be different.

To do this for all plugins/themes, we'd need to only build the ZIPs once and store them in a location all the servers can access. Or dedicate downloads to a single server or set of them.

#5 @dd32
3 years ago

I seem to recall something around April or so when mdawaffe or nacin remarked that plugin md5s from different servers didn't match

Yep, but I don't think this is a problem, It represents the MD5 of the content of the request, not a canonical MD5 of the package.

#6 @bgermann
3 years ago

To address the different ZIP timestamps one could also set the SVN use-commit-times option on those machines.

#7 follow-up: @Otto42
3 years ago

That wouldn't work in our case, because while svn --config-option config:miscellany:use-commit-times=yes export http://whatever does indeed preserve the commit times for the individual files, plugins and themes in the ZIPs are put into a directory (named with the plugin slug) which is created at the time of the ZIP. That directory will still have the current time for that particular server.

Note that this would not be an issue with core downloads, those ZIPs are built at release time and manually committed, they're not built by the system on-the-fly.

Last edited 3 years ago by Otto42 (previous) (diff)

#8 in reply to: ↑ 7 @bgermann
3 years ago

Are the build scripts publicly available? I search for them but did not find any. I would love to have a uniform download and would supply a patch for it.

#9 @Otto42
3 years ago

Not for plugins and themes yet, no.

#10 @bgermann
3 years ago

You could get the last commit date of the branch/tag which you are generating the zip file for and set the directory's date with touch -d or touch -t.

#11 @Otto42
2 years ago

#1410 was marked as a duplicate.

This ticket was mentioned in Slack in #meta by sam. View the logs.


23 months ago

#13 @ocean90
23 months ago

In [dotorg11195]:

Generate consitent plugin/theme ZIP files regardless of which server or time the zip is generated at.
This change causes all zip files to have the same hash, as long as they're generated from the same SVN revision.

@dd32, @Otto42: Is this something we can implement now?

#14 @dd32
23 months ago

Yes, we should be able to implement this now thanks to [dotorg11195]. I hesitated after that commit, as I wasn't fully sure if it would have the intended effect, we should do a survey of the zips to check they all match and then go for it.

#15 @ocean90
23 months ago

Note: [dotorg11195] applies only to new builds, plugins with no recent build still can have different packages.

#16 @dd32
23 months ago

As the MD5 won't be generated on-the-fly and only at package generation time, it'll only be for packages built after when it's enabled.

As part of the process I'll look into having the zips rebuilt where needed.

#17 @dd32
23 months ago

To summarize, when creating a zip file on-the-fly, there's several things you need to do

  1. svn export will set files to their last-commit-date, but directories to NOW(). Update directories timestamps to a known date (I chose the timestamp of the last commit to that theme/plugin).
  2. When copying files between directories, use --preserve=timestamps to preserve the timestamps rather than having them set to NOW().
  3. Using zip specify the -X parameter to strip filesystem-specific things such as user/group's and access time from the zip, as these are not needed and may vary.
  4. When creating the zip file, zip will add the files in the order it finds them on disk, however even if you specify a file order via STDIN, it'll add them in a seemingly random (and system dependant) order. Files need to be sorted and added individually to ensure that it's consistent.

The theme zip files appear to be being generated consistently now, plugin zip generation needs #4 applied to it still.

#18 @bgermann
9 months ago

  • Component changed from General to Plugin Directory
  • Milestone set to Plugin Directory v3 - Future

#19 @tellyworth
7 weeks ago

  • Keywords 2nd-opinion added

#20 @Otto42
11 days ago

This should definitely be doable now with the changes made to the ZIP building process.

I believe the original nginx rules mentioned should still work. Do we need to ping systems about it?

#21 @dd32
11 days ago

With the more recent changes made to downloads, this is no longer as easy as it previously was.

ZIPs are now reverse-proxied to a SVN server, and the server responding to the request no longer has access to the file to generate the MD5.
We'd probably have to cache ZIP md5's in the middle somewhere.

Personally I'm -1 on implementing anything here now, and would instead prefer to see the next step after Content-MD5 being implemented, a signed hash of the zip instead.

Note: See TracTickets for help on using tickets.