Release confirmations: users are seeing update notifications before an update is available

[barry.hughes]

We recently enabled release confirmations. Unfortunately, after committing a tag to SVN (but before we update the stable tag), users are seeing notifications that the new version is available. In fact, it is not.

This has happened over a number of recent releases (for WooCommerce) and, during one of those releases, we were able to replicate seeing the update notification ourselves on a staging site (see attached screenshot). We've been in touch with plugins@… and on ​Slack, but it felt like it was worth logging a ticket here—as this is causing confusion and eroding trust (some customers believe we have shipped a release, only to quickly 'pull it', which isn't what is happening).

I'll copy here the summary shared by my colleague in Slack, for easier reference.

Hey team! A few days after having enabled release confirmation for the WooCommerce plugin, we started noticing that committing a tag to SVN would not only create the pending release but also update the plugin version right away. I'd like to share some notes on this behavior and would appreciate your guidance. Happy to move this to a different channel or submit to Trac if that's preferred.

• Timing seems to indicate this started happening after this change: specifically that, if RC is enabled, $version gets overwritten with the tag name (L171) and then persisted as the version to plugin metadata (L457).
  • This, in particular, results in the plugin page displaying a version that is not the sable one as "Version" and also the update being offered to WP sites. Luckily, the download link in the plugin_information response still uses the stable tag, so even if someone "performs the update" it doesn't truly install a non-stable version.
• Our workflow might be a bit different in the sense that we commit the SVN tag and trunk together when releasing a new version, but only update the stable tag after a bit.
  • Despite this, we feel that committing a tag shouldn't necessarily make it the latest version immediately (more-so if it's not the stable one), and that wasn't the case in a previous iteration of the RC feature.
  • In our multi-step process, once we finally update trunk with the new stable, the import process takes care of returning things back to normal (as $version no longer gets overridden and it uses the version header in the stable tag).

The above is my take after having looked at the code for plugin-directory in the meta SVN repo, but I might be missing something due to unfamiliarity. Appreciate any feedback you could share. Thanks again!

[...]

I believe things go back to normal as soon as the release is confirmed (regardless of stable tag in trunk) because RC approval triggers import_from_svn() again but this time the release already exists and so the code doesn't enter the if() branch where $version gets overwritten. Instead, it updates meta with version from the headers in the stable branch, which is ok.

I hope the above gives some insights into what might be going wrong. Let us know if we can provide any further information—release confirmations are a great idea, so we're keen to seem them working well.

[barry.hughes]

9.1.2 update notification, which appeared *before* 9.1.2 was marked as stable.

[dd32]

In 13958:

Plugin Directory: Rename a variable, to see if this has caused the public version to change before a release is confirmed.

See #7742.

[dd32]

Hi!

As explained previously; the issue with 9.1.2 was indeed a bug that existed for a few hours, but that was resolved. The WooCommerce Beta Tester plugin also caused confusion around the same time with the team. As previously stated, any data that you've got with 9.1.2 that you've reported can't be relied upon, as it was 100% a timing problem.

The follow-up tagging of the 9.2.0-RC's doesn't appear to have any problems, and the WordPress.org data doesn't show 9.2, nor did it release as stable.

I was provided with a screenshot that suggested *9.1.4* was released before it was confirmed; but I can confirm that the data was not imported and that no 9.1.4 code was ever included in ZIPs before it was confirmed.

I do believe there may have been an issue where the publicly shown version number increased, based on the above commit, where it's quite possible that $version was overwritten with the newer version.
If that was indeed the issue; that would explain 9.1.4, and why the version shown on WordPress.org would've had the new version, but the ZIP and all other data would've reflected the stable version.
The same problem should have happened with 9.2-RC however if that was the case, and as far as I know, didn't.

I've been unable to duplicate this in testing, but in theory, that variable overwrite combined with specific commit steps (adding the new tag, changing trunk files, but not changing the stable_tag, all in a single commit) could likely have been the culprit here.

[barry.hughes]

Thank you for the update. We plan on shipping a new release next week, so will definitely watch for any further instances of this (but hopefully we're all set) 🙂

[jorgeatorres]

@dd32 Thank you for looking into this. We released both WooCommerce 9.2.0 and 9.2.1 this week and the version didn't change anywhere until we changed the stable tag version, so this seems to have been resolved now. Thanks again!

[dd32]

Thank you @jorgeatorres I'm going to close this as fixed based on that and the above comments.

Please feel free to re-open this and ping me directly if you see anything remotely similar!