limit "Tested up to" plugin header to current stable release

[littlebizzy]

Perhaps time to enforce an accurate (maximum) "Tested up to" header for plugins in the WordPress.org directory, to avoid confusion and misinformation.

Specifically, if plugin authors input a "Tested up to" version of WordPress Core that is higher than current stable version, the WP.org directory could overwrite search results and plugin profiles.

Example: ​plugin author claims to have tested a plugin up to WP 6.0, which does not exist yet, even in beta or bleeding edge repositories. In such cases, WP.org might alter this to display as "4.9" (the latest stable version).

Ref: ​https://make.wordpress.org/plugins/2018/02/13/not-updated-in-warning/

Can I put in 5.0 as a version? Yes, but it won’t do what you think it does. That is, you won’t show as compatible with 4.9. Don’t try to be clever on that one.

Ref: ​https://developer.wordpress.org/plugins/wordpress-org/how-your-readme-txt-works/

Make note that the Tested up to field ignores minor versions, as plugins shouldn’t break with a minor update. This means you only need to define the major version it is tested against and the WordPress.org plugin directory will automatically add the minor version for you. This should be numbers only, so ‘4.9’ and not ‘WP 4.9’

Or better yet, since such plugin authors likely aren't actually testing their plugins for the versions they claim, WP.org could display such "Tested up to" instances as simply "N/A"

[littlebizzy]

example of exaggerated "Tested up to" header

[Otto42]

Putting a version in that field which is higher than the current maximum actually actively hurts your search ranking. So by and large, the problem tends to solve itself.

[littlebizzy]

As mentioned, it goes far beyond just search manipulation concerns.

The vast majority of WordPress users don't keep up with all the release info, versions, etc so it is likely confusing them, besides just the glaring appearance of incorrect metadata.

Compare these with e.g. mobile app stores, which never allow apps to just "invent" new SDKs, etc.

Perhaps a few other people could offer their opinion before Otto shuts this down...

[jsmoriss]

Plugin authors received an email a while ago about updating their "Tested Up To" for 5.0. Should we not be using 5.0, even if the plugin is tested with the latest RC?

js.

[dd32]

Replying to jsmoriss:

Plugin authors received an email a while ago about updating their "Tested Up To" for 5.0. Should we not be using 5.0, even if the plugin is tested with the latest RC?

You should be doing exactly that IMHO, as you've tested with 5.0.

Some authors do however use exact versions ie. 5.0-RC1 and update them at release time, but unless your plugin has a developer audience or is likely to break, I wouldn't do that.

---

As stated by @Otto42, Having a 6.0 tested up to ranks you much worse in search, it's effectively like saying "I'm tested with 4.0" when the current is 5.0. Heavy penalty in search results, possibly worse than not including a version.

That all being said, changing how we import the number seems like a good idea to me.
If it's higher than whatever trunk currently is, it can be safely discarded.
While we're at it, we can also discard anything like PHP 5.2.x, WordPress 4.9, BuddyPress 1.5 too (That's a real example).

Looking at the raw data for published plugins, it'd only affect ~50 plugins who have future tested-up-to versions:

tested count
9.9		20
5.1		19
6.0		5
9.1		3
5.5		3
5.6		3

Those marked 5.5/5.6 I'm going to guess are actually PHP versions and the 9.9 is obviously a "I don't want to update this".

[dd32]

In 7874:

Plugin Directory: Readme: Validate the Tested up to and Requires at least fields contain a WordPress version value.

This commit will strip Tested up to/Requires at least values which are invalid versions, or higher than trunk/master - For example, currently that's 5.0, a value of '6.0' will be ignored, and 'Tested up to: PHP 5.2.4' will be also be ignored.

Fixes #3936

[dd32]

Re-opening after [7874] for possibly re-parsing of some of the plugins affected above with 5.1+ values as their Tested Up To.

Since we use both the Tested up to and Requires at least values in API's and other locations I also applied a similar validation/sanitization over the Requires at least header

[Otto42]

Mmmm.. not really a big fan of this approach. I would rather there be consequences of using invalid data. By ignoring it or overly sanitizing incorrect data, then this eliminates that penalty in the search rankings we talked about above.

If we want to hide such invalid data from the site+API, then that's acceptable, but having bad data should be a valid cause for bad results. Plugin authors should strive to get it right.

[dd32]

Replying to Otto42:

Mmmm.. not really a big fan of this approach. I would rather there be consequences of using invalid data. By ignoring it or overly sanitizing incorrect data, then this eliminates that penalty in the search rankings we talked about above.

Having no tested up to is a penalty, a fairly large one. Not only does it mean that they'll get ranked down in the search results (IIRC, far more than just having it set to 10 versions away from current) but they'll also get "Unknown Compatibility" warnings inside WordPress on the install screens and update screens - where as previously they got a great big "All good! Compatible!", that'll also cause more support requests asking if it's compatible (Yes, that does happen).

The sanitisations done can be removed if wanted, but they add so little overhead and protect developers who thought they were doing it right, but weren't - as we never enforced any form of sanity in the fields.

I'll pull a list of affected plugins today, along with some information about each and you can decide for yourself if it's really a problem.

[littlebizzy]

Having no tested up to is a penalty, a fairly large one. Not only does it mean that they'll get ranked down in the search results (IIRC, far more than just having it set to 10 versions away from current) but they'll also get "Unknown Compatibility" warnings inside WordPress on the install screens and update screens - where as previously they got a great big "All good! Compatible!", that'll also cause more support requests asking if it's compatible (Yes, that does happen).

The sanitisations done can be removed if wanted, but they add so little overhead and protect developers who thought they were doing it right, but weren't - as we never enforced any form of sanity in the fields.

This all sounds as perfect as can be.

Dishonest, lazy, or irresponsible plugin authors get penalized in search results, while end users are protected from confusion and misinformation. Organization and clarity is also improved... and Unknown Compatibility errors will encourage end users to ask plugin authors for answers, so the whole thing self-regulates.

Overall it should create more "honest" competition in the directory.

[dd32]

Replying to dd32:

Replying to Otto42:
I'll pull a list of affected plugins today, along with some information about each and you can decide for yourself if it's really a problem.

As promised, here's a list of those plugins:
​https://docs.google.com/spreadsheets/d/e/2PACX-1vRs_2lCNMdrFT460rmbkvg0VHEHOLusjx6N8EEtVM7ZqpEk3BGctnG_y2kJywknSjGj4ur6G6Hrz8MP/pubhtml?gid=1773978482&single=true

• Focuses on tested up to only
• Number of plugins are Tested up to: 5.1 which is excluded by the committed code, but seem intentionally set as such.
• Most of those with >7 are old plugins (which get ranked down heavily already) and by a limited number of authors
• A lot of the plugins with invalid versions can be imported correctly after this change, as it strips common inclusions in the field
• Adding [ '.*', '.x' ] to the strings to remove .would increase the number imported correctly significantly
• A lot of plugins want to specify the version of WooCommerce they're tested with, which is great, but it's a shame this field doesn't really work for that (other than being exposed in the WordPress.org UI as-is). It's a limited number of authors though.

Upon reviewing this list, I'm happy with the code and see no reason to change the approach.

[casiepa]

@dd32
For the tested up to, I would allow the version to go up to the current dev version but not higher.
The current dev can be found on ​https://api.wordpress.org/core/version-check/1.7/?version=100.100 and would give something like 5.4-alpha-47086, so the 'Tested up to' could be 5.4, but not 5.5 or 6.0

That would avoid having to calculate the +1 or +0.1.

[dd32]

Replying to casiepa:

@dd32
For the tested up to, I would allow the version to go up to the current dev version but not higher.
The current dev can be found on ​https://api.wordpress.org/core/version-check/1.7/?version=100.100 and would give something like 5.4-alpha-47086, so the 'Tested up to' could be 5.4, but not 5.5 or 6.0

That would avoid having to calculate the +1 or +0.1.

Calculating it as $max = LATEST_RELEASE + 0.1 is simple enough and it doesn't need anything complicated. Setting it to +0.1 affects search ranking negatively (It's the same as setting it to LATEST - 0.1)