Phased releases and roll-outs of plugins

[matt]

It's very common in other app stores and marketplaces to support a method that allows developers to roll out a release to a random subset first, gather feedback, and then deploy it to 100% of users as they gain confidence in the stability and reliability of the update. This isn't a replacement for beta plugins and QA, but rather a way to gather feedback from a wider range of sites and hosting environments.

Some prior art from iOS and Androd:

​https://support.google.com/googleplay/android-developer/answer/6346149?hl=en

​https://developer.apple.com/help/app-store-connect/update-your-app/release-a-version-update-in-phases/

Apple has a fixed percentage increase over 7 days, and Google makes you set everything manually.

This has been something I've heard a lot of requests for from plugin authors, particularly the larger ones. It would be excellent if we could support this in our directory and tie it into a general theme of plugin ecosystem enhancements in 6.9.

(We should do this for themes too, but I think plugins are a priority.)

[davidperez]

I support this idea, especially since some authors have been requesting access to beta code in SVN. I don’t think that’s ideal, as SVN should always contain production-ready code. With this approach, however, the final version will be more stable and reliable.

We could implement it in different ways — for example, by rolling it out to a percentage of users or triggering it every X hours.

[Otto42]

#5805 was marked as a duplicate.

[Otto42]

Implementing it by percentages is entirely possible, and we have actually already done that for core. However, to do something like this for plugins, we would need some kind of user interface to make it available. It would have to be on a tagged version level, possibly hooked in with the existing release confirmation system.

[dd32]

allows developers to roll out a release to a random subset first

I can see a major confusion point for end-users here, when they see a plugin update on Site1, and not on Site2. Especially a problem when it's resolving a bug they're experiencing on both sites.

I would suggest that staged rollout should be specifically only for auto-updates, which while I don't have the data to backup my opinion, I believe accounts for the majority of plugin updates today.

Combining that with improved messaging in the core UI of when a site should update to the plugin (ie. the API says "Update to v2.0 either by user clicking Update, OR automatically in +29hrs") I believe that concern would be adequately resolved.

gather feedback

I feel that we're currently not in an ideal position to fulfil that need. Support forums are the obvious place for feedback, but I don't feel that most end-users (and authors) make the best use of it.
Ensuring that we have a good feedback loop for both end-users and plugin authors would be needed to make the most of this.

[matt]

Just for auto-updates to start is fine.

You're completely right about feedback mechanisms. I think the only automated one we have is a callback if the upgrade fatals and we roll back? I know we have that for core, is that also something we can expose for plugins on the stats page?

Plugin authors also have their normal support mechanisms, and a significant percentage of people come to .org to provide feedback or leave reviews.

If there were a way to rate and review without leaving wp-admin, that would also be amazing for plugin devs, and lower friction than our current flow.

What are first steps and how can we break this down?

[ariel.k]

Thanks @matt for starting this proposal.

As a plugin developer with millions of active installs, we’re really excited about this idea and would love to be involved. Right now the plugin repo has ~70 plugins with 1M+ active installations, and we think that with their wide reach they also could benefit greatly from a more controlled rollout configuration.

I don’t know exactly how things work behind the scenes on wp.org, so I can’t weigh in on the technical side. But we agree with @dd32 that starting with auto-update users sounds like the right move, and we can then see how that goes.

About the percentage-based rollout that @Otto42 mentioned, I think it’s also a good starting point. I’d suggest adding a few more options, inspired by the Apple and Android rollout processes.

For example:

1. Immediate rollout - This would be the default. Best for patches, security fixes, or anything small and urgent.

2. Balanced rollout - For regular major or minor versions that are well tested but still might have some regressions: Day 1: 25% > Day 2: 50% > Day 3: 100%

3. Phased rollout - For high-impact changes, major refactoring, new APIs, or UI overhauls: Day 1: 10% > Day 2: 25 > Day 3: 50% > Day 4: 100%

The idea is to finish the rollout within one work week. So if a release starts on Monday, it should reach everyone by Friday. That way, problems don’t spill into the weekend.

I’m really excited about how this approach can help improve stability and reliability, especially for end-users.

Thanks

[illuminea]

Hi! Miriam from Elementor here. Just following up to @arielk's comment above - we'd love to help move this concept forward. What next steps can we take?

[azaozz]

Another big +1 for this! As a plugin author that has maintained plugins with millions of active installs for many years having some peace of mind at release time would be really helpful. Would be great to be able to detect if something in the new release may have gone terribly wrong without breaking the sites of millions of users :)

Update to v2.0 either by user clicking Update, OR automatically in +29hrs

Yep, sounds good. Even maybe can explain it as "slow updates" option, to both plugin authors and WP users/site admins.

Thinking about the mechanics: using percentage may not be ideal. Plugins with 5-10k of active installs would get less than 500 early updates. But plugins with many millions of active installs will have something like 50-60k, or even more. Seems that would defeat the purpose to some extend.

Perhaps having a threshold would be better? Something like 3-4k seems sensible? This number would be enough to detect if something in the release has gone really wrong, and fixes are needed urgently, and at the same time will protect the bulk of the plugin's users from having the same problems.

Also thinking it would be pretty nice if the ability of plugins authors to pause and resume a release is stand-alone. So the authors can react even if phased (slow) release is not used, but there are a lot of complains in the plugin's forum that something's wrong, sites are down, etc.

[smub]

This is a great idea. I agree with @dd32 that it should be for auto-updates only. When user requests an update by clicking a button, they should get the latest version because that's what they're expecting.

For auto-updates, I would say a good approach would be split over 6 days: 1, 5, 10, 20, 50, 100% - this would be good for large plugins and also major updates.

For plugins with less than 100k installs and / or minor updates, the phase can be shortened to 3 days: 20, 50, 100%.

A bonus would be to allow plugin authors to choose a shorter period for minor releases vs. major.

Another thing that would be helpful for plugin authors here would be more analytics in the view to see how the rollout is progressing and such - perhaps in a simple dashboard.

[dd32]

In this comment, please assume: Site is running version 1.0 of a plugin, version 2.0 is just released with a phased rollout, and if there's a bug is found, version 2.1 is released.

After looking further into this, I ran into a few shortcomings that has resulted in my suggestion in the earlier comments being not possible, or with some caveats

Firstly:
How to bucket / segment the sites.

The most obvious way to segment the sites is to extract the sites domain from the update-check HTTP user-agent, hash it, and convert it to a 0..100 number. That is the simplest and works relatively well for other places we bucket sites on WordPress.org (eg: Core auto-updates throttles auto-update availability during excessive network load)

There's a shortcoming here though, a site will always be in the same bucket of sites - always at 8% for example.

Instead of hashing the domain, we could hash the entire user-agent header. That would result in the bucket changing each time they update core. 8% one time, 78% the next version. That would be ideal, as the user is not always the "beta tester".

However, that means that sites that are in the first 1% are always getting the first updates for all plugins using this update strategy.

The ideal solution is therefor to calculate the percentage as:

percent = convert_hash_to_percentage( hash( site-domain-name + plugin-slug + plugin-version ) )

which results in the percentage always changing, but consistent for any specific plugin-update.
We'd have to exclude the WordPress version (and the entire user-agent header) as percentages would otherwise change when core updates are occuring. Including the plugin-version seems best to ensure a user isn't always the canary, but does result in their position in the update queue to change version-on-version, which causes an edge-case below.

Perhaps having a threshold would be better? Something like 3-4k seems sensible? This number would be enough to detect if something in the release has gone really wrong, and fixes are needed urgently, and at the same time will protect the bulk of the plugin's users from having the same problems.

Using a count threshold could be better. However, we'd have to store the number of sites we've offered each update to (and noting: We'd have to store a site identifier, as WordPress may ask for an update check literally every 5 minutes). Storing/Counting the number of sites that have updated to a version wouldn't be relevant, as there could be 10x that number who know about the update who are currently sitting on it until their auto-update time comes around again (See the timing section later on)

With the way plugin updates are currently, it's an all-or-nothing approach.

WordPress currently only accepts a single plugin update payload, which doesn't have any real ability to configure it.

My original intention here was to delay implementation of this into WordPress 6.9, however, we still need to support WordPress 2.8 ~ 6.8 clients anyway so we have to take those into account. It would be unexpected to a plugin author that this only applies for WordPress 6.9+ sites IMHO.

The major shortcoming of the existing WordPress plugin update check is that we currently cannot provide different details for auto-updates vs user-initiated updates.
We also cannot tell WordPress "Here's an update that the user can install, but DO NOT auto-update to it.".
As a result, auto-updates-only is currently not possible with stable versions of WordPress.

What can we instruct WordPress to do?

The WordPress.org plugin update API can return one of 4 states:

1. Plugin XYZ is not hosted on WordPress.org
2. Plugin XYZ is hosted on WordPress.org, and you're running the latest version: 1.0
3. Plugin XYZ is hosted on WordPress.org, and you're running an outdated version: 1.0. Version 2.0 is the latest.
4. Plugin XYZ is hosted on WordPress.org, you're running an outdated version (1.0 vs 2.0) BUT automatic updates (both user-initiated and background auto-updates) are not available for this plugin.

WordPress needs to know if the plugin is hosted on WordPress.org, so we have to return one of 2, 3, or 4.

By returning 4, WordPress will display a notice such as the following:

There is a new version of XYZ available. View version 2.0 details. *Automatic update is unavailable for this plugin.*

By clicking the View version 2.0 details you'll see the plugin detail modal with the Update now button, which will fail with an error of Update failed: Update package not available..

As a result, item 4 is not advisable, and the functionality mostly exists to support cases where the plugin is not hosted by WordPress.org and the update provider is not offering an update (but knows there's an update available).

That leaves us with options 2 and 3.

The simplest option forward here is, when 2.0 is released that we say:

• (2) Site not included (yet): You're running version 1.0, there's no updates available
• (2) Site not included (yet): You're running version 0.5, there's no updates available *(It's not the latest version, but because they're not included in the most recent update group yet, it's all we can tell them - we can't really store previous version data here, without causing additional awkward edge-cases)*
• (3) Site included in phased rollout: Hey version 2.0 of this plugin is out! Update now!

That's not a great user experience, as a user is unable to update a site even though they know the update is available on another site. But we don't (in WordPress 6.8) have another choice.

However, edge-cases are rough here, Let's say at the 25% rollout point a bug is discovered in the plugin, what does the author do?

• Rollback: The author sets 1.0 as the stable release again.
• Fixed Release: The author releases 2.1 to fix it.

In the rollback case, those running v2.0 *will not be downgraded* (and IMHO: shouldn't be), and if the author takes a week to move to fixing it, they'll continue to run the buggy version. That's highly likely acceptable to us.

In the fixed release case, assuming they use the same rollout strategy. Going back to the percentage above and the canary case.. that means that the bucket of users who receive the update first will be different to those who updated before - in other words, Site ABC was the FIRST site to update to version 2.0, but now there's a chance they'll be the millionth site to update OR the LAST to update to version 2.1.
I see this as just an acceptable edge-case. We could do logic like "If they received the 2.0 update, update to 2.1 immediately" but that would require storing additional data about the site and what it had been offered.

Another edge-case here is that if the site was running version 0.5 of the plugin, and saw an update to 1.0 is available, having the site attempt to update to 1.0 MAY fail if version 2.0 is released at the same time. This is due to race conditions between the WordPress.org API, the transients in WordPress, and the way the user-interface uses that data.
This is just an edge-case I'm documenting here, that I don't believe will realistically ever be an issue.

Install Count

Phased updates using the hashing method of bucketing is likely to result in at least one site always being in a bucket, however, smaller plugins will inherently have less sites per bucket. Combine this with the fact that many sites do NOT update their plugins, and do not have auto-updates for plugins enabled, means it could be a while for smaller plugins to get an update.

I would suggest this feature needs to be limited to >10k active installs for plugin author simplicity. 10k is "too small" for usefulness in many respects, but choosing a larger count is likely to exclude some plugins that would benefit from it.

Update Timing

I've modeled a few different strategies for update availability, but there's a few major kinks in how WordPress plugin update-checks operate.

WordPress has a decoupled notion of how auto-updates operate, and how that interacts with WordPress knowing about a plugin update.

Consider this:

• WordPress core/themes/plugin update checks run twice daily
• WordPress auto-updates only run during the core update check

When you visit the WordPress plugins page, it might find out about the plugin update immediately (as it just performed the update check) but it won't auto-update the plugin until the twicedaily core update check runs.
In those situations you'd see something like:

There is a new version of Plugin XYZ available. View version 2.0 details or update now.
Disable auto-updates
Automatic update scheduled in 11 hours.

As a result, any phased rollouts for plugins realistically has a large lag time. This increases the overall length of time that updates need to be released at. We're not talking "5% in the first hour, and increase 5% every hour after that" We're talking "5% in the first 12 hours, and 5% every 12 hours thereafter".

We could:

• Schedule auto-updates to run immediately once it knows there's an update (of any kind) available and the next scheduled auto-update is not for hours.

I've graphed some initial thoughts of what I would expect as the types of rollouts buckets, I personally only feel that the cautious (5d) and slow (3.5d) strategies are worthwhile when you consider the lag above.

Another thing that would be helpful for plugin authors here would be more analytics in the view to see how the rollout is progressing and such - perhaps in a simple dashboard.

What data would you want to see here? Noting that we can't realistically provide the number of sites updated (or a total count of sites), we could show "60% are running this version" but that's not useful if the other 40% are running a version 10x older. "5x more people are using this than LAST_RELEASE" also not super useful IMHO.

What I think you're asking for is a progress meter, but we don't really know what the upper-bound of that is, so we can't really calculate the progress. The upper bound is NOT the active install count.

Feedback to author

Finally, having a ​canary-in-the-coalmine is not useful if the workers are not watching the canary or if the workers are in a different mine all together.

I alluded to this earlier, and elsewhere suggested that reviews/ratings could form part of this.

Matt: If there were a way to rate and review without leaving wp-admin, that would also be amazing for plugin devs, and lower friction than our current flow.

Bad reviews are far more likely to come from an intentional change in behaviour than bugs IMHO, where a phased rollout would likely not prevent it. As such, i've split this into #8031 as it'll also require some significant work on how forums reviews work, how WordPress.org account registration works, and how spam would be dealt with.

But to focus on my main concern here: If a plugin is rolled out to 5% of users, and it's a bug that only 10% of users run into, then that's 0.5% of users who are both affected and updated, and then consider that maybe only one-in-ten would actually go to the effort of reporting it, then we're at a tiny likelyhood of issues being reported for some plugins. For multi-million-install plugins this isn't likely an issue as the issue would likely be reported fast.

But to use my experience of monitoring the forums, when a plugin releases a slightly bad update it can generate maybe 3 threads on the forums with a dozen people interacting with it.. and that's when it takes the author several days to respond to the issues.

I'm still fairly concerned that we don't have any reasonable ways for authors to really know that there's a problem. I'd like to hear more from Plugin Authors on how we can address that, or if it's really something we're not going to be able to begin to solve until after this feature has been used for a while.

See https://meta.trac.wordpress.org/ticket/8009

This rough code is not intended on being final, and is just my early experimentation of implementation.

[namith.jawahar]

Until there is a UI in place, we can probably use an entry in the plugin header

Rollout: [immediate, balanced, phased]

but without a feedback mechanism for plugin authors it wont be of much use

[bradshawtm]

As mentioned in several comments, for this to be effective we need some way of surfacing errors back to the authors.

If we go with the rollback option, perhaps we could capture the errors and offer to send it to WP.org, and then on the plugin admin page authors could see a rolling list of 1000 most recent logs submitted. But that introduces a vector for abuse and spam as well.

[tw2113]

How would roll-out priority be gauged?

If a plugin has opted into phased releases, would sites that haven't been actively touched in say 1 year+ be potentially first in line, leaving the available update in limbo?

or would sites that are at least relatively actively maintained be considered first?

[nerrad]

I'm still fairly concerned that we don't have any reasonable ways for authors to really know that there's a problem. I'd like to hear more from Plugin Authors on how we can address that, or if it's really something we're not going to be able to begin to solve until after this feature has been used for a while.

@dd32 Brief idea. I wonder if we could add some additional functionality to the WP Fatal Error Handler:

• If stack trace on error can reliably match the fatal error path to an installed plugin path...
• Match to a wp.org listing...
• Send a package to a new endpoint with the error details, masking any sensitive details, and/or send email to plugin author.
• Potentially make basic stats available to the plugin author on the count of fatal errors reported in the rollout with aggregate list of errors?

This might require opt-in for sites. Also, this won't capture all the relevant errors and would also have to account for potential false signals - for example, another plugin with a compatibility issue actually is what shows up in the stack trace.

[jazzs3quence]

Just wanted to jump on this minor-ish point really quickly:

If there were a way to rate and review without leaving wp-admin, that would also be amazing for plugin devs, and lower friction than our current flow.

This is a solution to a different problem. I wouldn't want ratings to (ever) be used for feedback about beta/pre-release/early phased rollouts of plugin updates unless reviews clearly indicate the version of the plugin that they are referring to. Users already leave negative reviews that are actually requests for support that they didn't make in the support forum.

I agree that making it easier to review/rate plugins in the WP dash is a good idea, but I don't want to conflate that with phased rollouts for plugins or a feedback mechanism for early releases of plugins.

[joshuaflow]

Much of this has already been mentioned but I'll add my 2 cents since a phased rollout is part of our release process. The rollout itself is extremely helpful when it comes to discovering bugs due to a broader range of plugins, environments, and settings than those used during testing.

A few items that we've found important for this rollout:

• Gradual release - Releasing to a small number of pools and gradually ramping up is very important.  This allows us to slowly uncover errors instead of crashing all the sites if a more severe issue is found.
• Rollback - When something more critical is discovered, we need a way to quickly revert those changes. Leaving sites affected with a critical issue for a length of time is not acceptable.
• Quick feedback - Timely feedback on issues is important in both addressing them and rolling back if necessary.

Some of the challenges we've encountered with these rollouts include:

• Reproducing issues - In order to be able to reproduce issues, we need as much environmental context as possible. This is much easier on your own testing infrastructure where you have full access to logs compared to issues that occur in the wild.
• Plugin conflicts - This point may be controversial, but when you encounter a critical issue, the onus to resolve the issue (seemingly) falls back to the plugin doing the rollout even when those issues may stem from other plugins incorrectly using hooks/APIs.  This tends to affect monolith plugins and those that are extended by other plugins far more often.
• Monitoring - We need to monitor proactively, which has come with a pretty steep learning curve.  Dashboards and specific error filtering are critical to diagnosing problems quickly.

[namith.jawahar]

A new dimension to the downloads graph can be a quick solution to reporting showing downloads vs errors, if its crosses a threshold, rollback the plugin on w.org, send notification email to plugin author asking them to rectify and upload an updated version.

[azaozz]

Replying to dd32:

Great way to put this together, thanks! I'd like to add few ideas/suggestions, hope they make sense.

The most obvious way to segment the sites is to extract the sites domain...
There's a shortcoming here though, a site will always be in the same bucket of sites

Thinking that a good way to implement this (at least at first) would be to keep it as simple as possible. Imho the main consideration here is that choosing which sites will get update notifications early and which will get delayed is not critical. Preferably it would be random, but no need to enforce only one try per site per day, etc.

In that terms think there is no need of buckets or storing any site identifiers. Only thing that needs to be stored is how many sites were actually updated. Seems this data would be coming from the downloads counter. In my mind perhaps the simplest way this could work is:

1. When a plugin update gets released and sites check for updates the wp.org API would offer an update as usual. Can also be throttled but not sure if it will be needed. Depends on how often the download counter is updated/checked, see below.

2. At update release time a number of initial updates is (pre)determined and stored in memory. This could be percentage based, or better a predetermined number. Download count of the new version is checked against this number every minute of so. Once the number is reached (or exceeded) the API would stop to offer the update.

3. If there is a problem with the release the plugin authors would be able to pause or suspend it. If no errors the API would automatically continue to offer the update (perhaps) 24 hours after the initial release.

I see many people here suggest more control and features for plugin authors, but perhaps starting simple would be best. Thinking we can come up with good initial defaults that will work satisfactory for most plugins. For example I agree that phased releases would not make sense for plugins with less than 10k active installs, perhaps even 20k. Another good default would probably be the auto-continue time set at 24 hours.

As mentioned in several comments, for this to be effective we need some way of surfacing errors back to the authors.

I've been maintaining quite a few plugins through the years and have always depended on the plugins support forums. It's true that the forums for plugins with many active installs are way more lively, but even 1-2 comments soon after a new release should be enough to draw the author's attention. Imho this can be considered an adequate way of notification for now.

I agree that it should be for auto-updates only.

The simplest way to do this would be to stop offering the update as soon as the number of downloads reaches the number we initially intended to update. Then the API would auto-continue to offer the update 24 hours later (TBD). Don't think it makes any difference whether the plugin was auto-updated or not.

I agree it would be nice to have "manual override" where the site admins will be able to update a plugin despite that the number of initial updates has been reached and the API doesn't offer updates at that time. Of course they will need to be informed about what they are doing. This would have to be implemented in WP. Any takers? :)

Instead of hashing the domain, we could hash the entire user-agent header. That would result in the bucket changing each time

Don't really think it matters much if the API skips offering the update on the first check by a particular site and offers it on the next. Not critical by any means.

Using a count threshold could be better. However, we'd have to store the number of sites we've offered each update

The (simplest?) way this may work is to have a pre-determined number of "early updates" for each plugin release, then check it against the download counter for the updated version (see above). That way it doesn't matter which sites we offered the update to. If they were updated we're all done and the site admins won't see the update notifications any more.

My original intention here was to delay implementation of this into WordPress 6.9, however, we still need to support WordPress 2.8 ~ 6.8 clients anyway so we have to take those into account.

Yea, backporting this may be difficult (if we want to allow site admins to manually override). If not, thinking the code would be the same for new and old versions of WP. I.e. it will be controlled by the API.

The major shortcoming of the existing WordPress plugin update check is that we currently cannot provide different details for auto-updates vs user-initiated updates.

Right. Thinking it would be helpful for the API to be able to tell WP (the client) to auto-update or not to auto-update. But seems this can be initially implemented without that functionality.

The WordPress.org plugin update API can return one of 4 states:

1. Plugin XYZ is not hosted on WordPress.org
2. Plugin XYZ is hosted on WordPress.org, and you're running the latest version: 1.0
3. Plugin XYZ is hosted on WordPress.org, and you're running an outdated version: 1.0. Version 2.0 is the latest.
4. Plugin XYZ is hosted on WordPress.org, you're running an outdated version (1.0 vs 2.0) BUT automatic updates (both user-initiated and background auto-updates) are not available for this plugin.

By returning 4, WordPress will display a notice such as the following:

There is a new version of XYZ available. View version 2.0 details. *Automatic update is unavailable for this plugin.*

By clicking the View version 2.0 details you'll see the plugin detail modal with the Update now button, which will fail with an error of Update failed: Update package not available..

Ugh, a bug. Needs fixing.

As a result, item 4 is not advisable, and the functionality mostly exists to support cases where the plugin is not hosted by WordPress.org and the update provider is not offering an update (but knows there's an update available).

That leaves us with options 2 and 3.

Right, in the above idea the API will be returning 3 until the pre-determined (percentage|threshold) number is reached, and then switch to return 2. Once manual override is implemented in core the API can include a flag to tell WP that there is an update that is part of a phased release, and the particular site was skipped. Thinking it would be okay for only new versions of WP to have this functionality.

I would suggest this feature needs to be limited to >10k active installs for plugin author simplicity. 10k is "too small" for usefulness in many respects, but choosing a larger count is likely to exclude some plugins that would benefit from it.

Frankly my gut feeling was 50k active installs, going down to 20k after couple months, but I see your point :)

As a result, any phased rollouts for plugins realistically has a large lag time.

Seems this will be true for plugins with less active installs. Plugins with lots of installs (>1m) will probably get a lot of manual updates too so the rollout will be quite fast.

I've graphed some initial thoughts of what I would expect as the types of rollouts buckets

Sounds good but maybe can be considered as a later update/enhancement? Seems that the important functionality of phased releases can be achieved relatively simply, and buckets are not strictly necessary?

Another thing that would be helpful for plugin authors here would be more analytics in the view to see how the rollout is progressing and such - perhaps in a simple dashboard.

What data would you want to see here?
...
What I think you're asking for is a progress meter

Again, sounds really good but perhaps not in the initial release?

I'm still fairly concerned that we don't have any reasonable ways for authors to really know that there's a problem. I'd like to hear more from Plugin Authors on how we can address that, or if it's really something we're not going to be able to begin to solve until after this feature has been used for a while.

Thinking that monitoring the forums after a new release is a fairly standard practice. Right, there can be better ways, but they all require the plugin's users to act. Maybe having a "Report a problem" button on each plugin row in the Plugins list table would help somewhat. That can also include the output from site health. But realistically this is a (separate) WP core problem.

[dd32]

@tw2113: If a plugin has opted into phased releases, would sites that haven't been actively touched in say 1 year+ be potentially first in line, leaving the available update in limbo?
or would sites that are at least relatively actively maintained be considered first?

Unfortunately WordPress.org has no idea about any given WordPress install - We don't know if it's been logged into, if it's a spam site, if it's updated a plugin, etc.

While we could look at the version of WordPress it's using, or the previous version of the plugin it's using (ie. Is it using a 1month old version of the plugin / last stable, or is it 5 years out of date) that's going to significantly complicate calculations and processing speed (The update-check API for example receives 10's of thousands of requests a minute, even 10ms adds up)

@nerrad: I wonder if we could add some additional functionality to the WP Fatal Error Handler:

I'd love to see this, and it would greatly help plugin authors and the stability of WordPress sites long-term.

This probably should happen regardless, and should be implemented on it's own merits, rather than attached to this feature.

I know some people will have concerns about privacy (even if it's opt-in) and access to the raw data though. Performing anything significant in a PHP fatal error handler has also been problematic in the past from a PHP perspective.

@joshuaflow: Rollback - When something more critical is discovered, we need a way to quickly revert those changes. Leaving sites affected with a critical issue for a length of time is not acceptable.

Updating the WordPress plugin updates system to allow a forced downgrade would be useful in certain circumstances, however we are limited in what we can do here IMHO.

When you're controlling the hosting environment you can rollback very easily.

When it's WordPress.org, we have to rely upon the fact that the WordPress site will check for new updates in 12 hours time for another update, and if we tell it something then, it'll (maybe) install that update anywhere from 5minutes to 12hours later.

In other words; there's no choice in it, the site WILL be running the aborted version for likely at least 12 hours, and potentially up to 24hrs before it performs a follow-up action.

For many plugin authors, they'd be able to release a patched version in that time instead. So much so that I suspect many authors would not abort a release until they knew what was wrong (ie. to prevent one persons plugin conflict stopping the release that works for a million others).

@namithjawahar: A new dimension to the downloads graph can be a quick solution to reporting showing downloads vs errors, if its crosses a threshold

The problem is what is "an error"? If we're talking PHP Fatal errors that's one thing, but we can't exactly detect Javascript errors, plugin conflicts, or other similar broken functionality.

If plugins could have a "health check" (similar to docker containers) that indicated they're operating correctly that might help here, but still often a plugin doesn't know it's operating badly - and that's the entire reason the bad plugin releases exist.

@azaozz In that terms think there is no need of buckets or storing any site identifiers. Only thing that needs to be stored is how many sites were actually updated.

The main reason I have against the counting method, is that while we can count based on ZIP downloads, or post-update pings, if we were limiting to say 5k sites to start with, that doesn't help that we've returned update responses to 10k other sites already that will perform their updates in the next 5 hours time.

In other words: WordPress does not apply plugin updates when it finds out about an update being available.

3rd-party tooling (Such as Managed WordPress hosts) however likely DO update as soon a plugin release is made, but I'm basing my thoughts on core WordPress, as these may change over time.

The counting could of course be at the update-check serving level "OK, I've told 50k sites to update, let's put the brakes on for an hour" but there's no guarantee that they'll update - that 50k might see 5k or 45k updates. I think this is what you might be getting at.
We're a little hamstrung by how to store that data, Memcache doesn't guarantee that the data will remain (and we sometimes have to reset it), and database writes are expensive. Memcache would be "good enough" for most people even if it reset, but would be frustrating to users and authors when 50% of the way through an update it reset back to starting over again.

Definitely solvable problem, but I personally prefer statistics, and it's what we've used on WordPress.orgs APIs for this type of bucketing for a long time in the core updates.

@azaozz: Seems this will be true for plugins with less active installs. Plugins with lots of installs (>1m) will probably get a lot of manual updates too so the rollout will be quite fast.

I don't have any data on the manual-vs-auto update information for plugins, but I don't think user-initiated updates within the first day or so are likely to be significant enough.
While they 100% occur, based on the data I'm looking at, I expect in the order of 99% of updates that would be affected by this would be automated in one way or another. The downloads-per-minute graphs are just too consistent to be anything else.

One thought is that we could hold back auto-updates for the first 24hrs and just let manual updates occur, users that update plugins are more likely to report issues and to actually be using the site. The problem with this suggestion is that we have no way of preventing a site performing an auto-update or signaling to it at present. We could add this to WordPress 6.9+ but again, what do you do about existing and yesterdays WordPress's.

@azaozz: Thinking that monitoring the forums after a new release is a fairly standard practice. Right, there can be better ways, but they all require the plugin's users to act.

That's my point :) Support forums ARE a good way to gauge things, but it is not foolproof AND we're talking about a very small percentage of users who actually create threads verses the number of installs a plugin has.

@azaozz: Maybe having a "Report a problem" button on each plugin row in the Plugins list table would help somewhat.

Originally that's kind of what I was thinking for with the Ratings/reviews from the plugins dashboard, being able to say "Nah this has broken my site, sad-face 1 star." but you're right, ratings aren't necessarily the important thing here, being able to report an issue and get it resolved is.

It might need to be considered whether Create a support thread is appropriate for the purpose though. It might be that Report a problem which feeds into anonymous aggregated plugin feedback would be better.. but at that point, you might as well collect positive feedback too and just call it Reviews...

[lopo]

At Yoast, we’re interested in exploring this, but in our case we’d need to make sure that the rollout of our premium add-ons can be properly aligned with the release of the free plugin. Keeping everything in sync is key for us to avoid version mismatches or unexpected issues for our users.
Happy to be part of the discussion and help move this forward!

[joshuaflow]

Replying to dd32:

Updating the WordPress plugin updates system to allow a forced downgrade would be useful in certain circumstances, however we are limited in what we can do here IMHO.

Agreed and this a good limitation of WordPress.org to call out. My comments are simply noting the items we've found as requirements for this system to work in the case of WooCommerce.

For many plugin authors, they'd be able to release a patched version in that time instead. So much so that I suspect many authors would not abort a release until they knew what was wrong (ie. to prevent one persons plugin conflict stopping the release that works for a million others).

I think this is true most of the time. There are 2 scenarios where this doesn't unfold well:

1. The fix takes longer than that window to resolve.

This can be worked around by customers manually reverting or by providing a patch release that reverts the offending code. The latter does mean more patch releases, which may or may not be cumbersome depending on the plugin's release process.

2. The issue is more widespread and affects more customers.

In the rare case where a critical issue is discovered in a larger number of sites (due to different hosting environments, plugin conflicts, etc) I don't think it's acceptable to leave a site non-functional for up to 12 hours.

Again, these problems tend to hit larger plugins more due to the number of plugins consuming their APIs, hooks, or endpoints in unconventional ways. This case may be the minority of plugins, so I don't want to block the discussion with this point, but I do think it would be useful (at least at the customer level) to more easily rollback one of these updates.

[websiteredev]

I want to echo the suggestion that this be made an opt-in feature, possibly something that users are prompted to agree to when they install the plugin. Or at the very least, make it a feature that users can opt-out of. I've worked with several agencies that have a large number of sites which receive minimal updates, which are not good candidates for experimental updates, while others may receive updates on a monthly or quarterly basis and would be better suited for something like this.

[azaozz]

Replying to dd32:

The main reason I have against the counting method, is that while we can count based on ZIP downloads, or post-update pings, if we were limiting to say 5k sites to start with, that doesn't help that we've returned update responses to 10k other sites already that will perform their updates in the next 5 hours time.

In other words: WordPress does not apply plugin updates when it finds out about an update being available.

The counting could of course be at the update-check serving level "OK, I've told 50k sites to update, let's put the brakes on for an hour" but there's no guarantee that they'll update - that 50k might see 5k or 45k updates. I think this is what you might be getting at.

Right, as the initial updates number is not critical was thinking it would be acceptable to have some variations, for example "5k means 4.5 to 5.5k". This would still allow for the functionality to work correctly, and will simplify the rest of the code/logic quite a lot.

For example, if we wanted to initially update 5k sites we'll probably want to send the update notification to 6 or 7k sites. That number will of course depend on the ratio between manual and auto-updates, and will continue to grow (slower) for the next few hours (because of the continuing manual updates on sites that use cached data from the API). Of course we'll be able to tweak the number of extras once this is released and we're able to monitor the updates flow.

We're a little hamstrung by how to store that data, Memcache doesn't guarantee that the data will remain (and we sometimes have to reset it), and database writes are expensive.

Perhaps a good compromise would be to store the data in Memcache and flush it to the DB every minute or so? (At flush time it can check if the new number looks okay, i.e. is not smaller than the DB number, and if yes it can be restored from the DB). This is again based on the premise that the numbers are not critical and some variations are acceptable.

Of course if this can be made to work similarly to, or even reuse some exiting code it would be very nice.

I don't have any data on the manual-vs-auto update information for plugins, but I don't think user-initiated updates within the first day or so are likely to be significant enough.

I don't have any data either but have monitored updates for a 2m active installs plugin few times before there were auto-updates in WP. It somewhat depends on the release time but generally (as far as I remember) the first 10k downloads happen in less than 10 minutes, reaching 100k in about 2 hours, and about 1m in 24 hours.

One thought is that we could hold back auto-updates for the first 24hrs and just let manual updates occur, users that update plugins are more likely to report issues and to actually be using the site.

Yea, this is a good idea.

The problem with this suggestion is that we have no way of preventing a site performing an auto-update or signaling to it at present.

Right, has to be fixed in core. The API should be able to set whether to do an auto-update. Thinking this would be a fairly simple enhancement.

Support forums ARE a good way to gauge things, but it is not foolproof AND we're talking about a very small percentage of users who actually create threads verses the number of installs a plugin has.
..
It might need to be considered whether Create a support thread is appropriate for the purpose though. It might be that Report a problem which feeds into anonymous aggregated plugin feedback would be better.. but at that point, you might as well collect positive feedback too and just call it Reviews...

Thinking more about this, you're right. A standard support thread may not be best. Also afaik one of the difficulties with that form of feedback is the requirement for the user to long in at wp.org as that may affect privacy, etc. it seems anonymous feedback would be better. That would not only solve any privacy related issues/questions but will also make it easier to use as the "reporter" will not need to log in.

I kind of imagine something like this: on clicking "Report a (plugin) problem" a modal opens with the site health info pre-filled. The user will be able to add few words to describe it. Submitting it would go through their WP install (TBD, but seems preferable not to be directly to wp.org; privacy, etc. and there can be filters for plugins to access it) and WP will send it through the API.

Then these semi-automated reports can be visible only by the plugin authors, not public so they are unattractive for spammers. Or perhaps we can try to aggregate some of the data and post to the plugin's support forum so other users are aware (if they ever look there..).

Thinking this is worth a core trac ticket at this point :)

[azaozz]

Replying to websiteredev:

make it a feature that users can opt-out of
...
not good candidates for experimental updates

Not sure there are any "experimental updates" here :)
This feature doesn't concern site admins. It is to let plugin authors have a bit more control over the release downloads cadence, and be able to slow down or pause the update notifications sent by the wp.org API.

From a site admin perspective this will be almost unnoticeable. This is how update notifications work in simpler terms:

• Currently the notifications depend on the release time and triggering of an update check from the site. I.e. if an update is released at 12:00 noon, and a site checks for updates at 12:01, it will probably be one of the first to show the new release notification to the site admins. This is totally random.

• When this is implemented the randomness remains pretty much the same. Sites that happen to check for updates soon after a release will still get the notification. The only difference would be that the plugin author will be able to slow down or pause the release notifications. So sites that happen to check few hours later may not get the notification on that check but on the next.

So people (site admins) that want to "play it safe" and wait for a day or two before updating plugins will be able to continue to do so. There will be no changes there :)

[dd32]

Replying to 24:

@websiteredev: [...] this be made an opt-in feature [...] users are prompted to agree to when they install the plugin.

In this case, nothing would need to be opt-in as the plugin releases are not intended on being "experimental" or unstable. It's the same stable release that the site owner would have to opt to installing. We're not suggesting forced beta updates for a segment of sites, or anything of the sort.

If an agency doesn't perform plugin updates (Either manually, or automatically) this won't affect them, as they won't update in the first place.

[dd32]

Replying to azaozz:

Right, has to be fixed in core. The API should be able to set whether to do an auto-update. Thinking this would be a fairly simple enhancement.

Oh, Turns out this is already a feature of the API, via ​#core50824 - I had completely forgotten this existed and it's not documented in the API code..
Unfortunately the flag is completely broken between WP 5.5-6.5 (despite being realised shortly after addition), and so it can only be used in WordPress 6.6+ ​#core52796

So yes, we can totally disable auto-updates for the first 24hrs of a plugin release for a significant portion of sites without any core changes and minimal WordPress.org changes.

I kind of imagine something like this: on clicking "Report a (plugin) problem" a modal opens with the site health info pre-filled. The user will be able to add few words to describe it. Submitting it would go through their WP install (TBD, but seems preferable not to be directly to wp.org; privacy, etc. and there can be filters for plugins to access it) and WP will send it through the API.

Then these semi-automated reports can be visible only by the plugin authors, not public so they are unattractive for spammers. Or perhaps we can try to aggregate some of the data and post to the plugin's support forum so other users are aware (if they ever look there..).

I think this is a reasonable approach, and mostly what I was thinking.
This could also be combined with the before-mentioned Fatal errors collection.
This is probably best as a separate Meta ticket though (@azaozz Would you like to write one up?), maybe prior to being a core ticket, as the majority of the work concerns WordPress.org.
I do have a ​wporg-experiments-plugin that was intended as a testing ground for this kind of WordPress <=> WordPress.org interoperability things.