Enhancement for Reviews/Ratings: Use a Rolling Time Frame for more current reviews

[cais]

Currently the reviews of plugins (and themes for that matter) go back to a static point where there is also a comment that was made with the review. This is helpful but I believe it would be more beneficial to have the review be on a rolling time frame, perhaps matching the automatic closing of support topics without activity (one year?)

Also to note, plugins that have not been updated in two years or longer will not appear in general search results. Given this idea, an old plugin with very strong reviews could be misleading to users with its quality and stability if used with current WordPress versions.

Given how rapidly plugins, themes, and the general WordPress environment in which they operate changes, putting the focus on relatively current reviews would ensure that they more accurately reflect current state and performance of themes and plugins.

[scottwyden]

This is a fantastic idea!

[Otto42]

Not sure I'd go for one year.

Old doesn't mean broken. WordPress strives for backwards compatibility, and the fact that lots of 5-6 year old plugins still work perfectly is proof of that. Yes, there is some breakage, but if a plugin is actually broken, we'd rather remove it from the listings instead of simply making everything "old" hidden.

The 2 year standard for listing has worked relatively well, but reviews don't necessarily seem to have that apply to me. A review is still a legit review, even if it's two years old. Making ratings "expire" doesn't seem to me that it would actually surface better results.

[cais]

I'm not saying the review was not legitimate when it was written, but if there have been updates and improvements to the plugin (or theme) after the fact that would have that review no longer be "correct" then a rolling time frame would help to improve the review (and rating) results.

I see current reviews as more relevant than dated reviews, especially for current plugins ... I'd even accept a rolling time frame dating back from the last release of the plugin.

[Otto42]

Well, we do show most recent reviews first, although I'd like to change that a bit and try to surface better reviews. Substance over date, sort of thing.

[cais]

Substance would obviously be the key; and I see the date as being relevant to how substantial the review is ... I'm not so sure they can be separated and still hold the same value.

[deconf]

I feel like counting latest X reviews will better reflect the current state of plugins. In my opinion "expired" reviews should stay in place but not count.

A review is still a legit review, even if it's two years old.

@Otto42 A user can always update his review. By updating it, he can make his opinion count again. The advantage is that his updated review will reflect the current state of the plugin, instead of a two years old one.

The ability of leaving new reviews or updating old ones also covers the case of old plugins. It really doesn't matter how old the plugin is, the reviews will always show its current state.

Counting latest X reviews may also help in cases where plugins with lots of positive reviews fall into the hands of wrong people or choose a wrong path.

My worry about counting reviews based on time frames is that in some cases the number of reviews could be so low that it won't really reflect the truth about a plugin.

[SergeyBiryukov]

Replying to cais:

Currently the reviews of plugins (and themes for that matter) go back to a static point where there is also a comment that was made with the review. This is helpful but I believe it would be more beneficial to have the review be on a rolling time frame, perhaps matching the automatic closing of support topics without activity (one year?)

As noted above in comment:6, a user can update their review at any time to reflect their recent experience.

Is there any other particular changes that should be done here?

[cais]

Replying to SergeyBiryukov:

As noted above in comment:6, a user can update their review at any time to reflect their recent experience.

Although my review philosophy is not in question, I'm fairly certain I have written at least one review in all my years working in and around WordPress although I have no way to easily find it since it has been more than two years since one was written and my "activity" report only goes back two years(?!)

Granted a user may be able to update their review and/or change their star rating (up or down as their want), making those reviews older than two years still be considered while not easily found with the current interfaces does present challenges thus supporting the point of using a rolling time frame.

That is not to say older reviews, especially for the example of utility plugins without an update in 5 or 6 years (as examples, such as noted above) are not valid or useful, they are more edge cases in this scenario; and, if I understand the search functionality of the plugin directory these same older plugins would not be found unless their specific name is searched. This would take them out of the equation as far as comparing current plugins offering the same functionality where many potential users have little more to decide with other than recent reviews and current star ratings.

[samuelsidler]

Replying to SergeyBiryukov:

As noted above in comment:6, a user can update their review at any time to reflect their recent experience.

While users can, anecdotally they very rarely actually update their reviews.

Is there any other particular changes that should be done here?

I think a rolling timeline makes sense. We might want to consider different timeline lengths, depending on active usage. For example, if a plugin has 1 million+ active users, a shorter rolling window might make sense, whereas a plugin with 10,000+ active users might get a longer rolling window (due to fewer reviews).

@joostdevalk and I discussed this recently and I know he has thoughts on it.

[joostdevalk]

@joostdevalk and I discussed this recently and I know he has thoughts on it.

To be honest, I think we should mimic the App Store model. If a plugin hasn't had updates for 3 years, a 3 year old review can still be valid. However if a plugin has been updated every 2 weeks, a review written 6 months ago might already be completely invalid.

So, I'd suggest we start recording a version number next to each review. We keep a record of the last X version numbers and only show reviews for those X versions. Of course this would allow plugin developers to game the system by doing X releases to get rid of 1* reviews. So maybe we should add some more... Anti gaming rules.

[samuelsidler]

Hm, good point. Perhaps a mix of releases and time? One thing to consider is that we don't necessarily know what version a user is reviewing when they leave a review, unlike the App Store.

[joostdevalk]

Replying to samuelsidler:

Hm, good point. Perhaps a mix of releases and time? One thing to consider is that we don't necessarily know what version a user is reviewing when they leave a review, unlike the App Store.

We should probably fix that too. Would fix a whole lot of sock puppeting if people could only review a plugin they had actually installed on their site...

[SergeyBiryukov]

Replying to samuelsidler:

One thing to consider is that we don't necessarily know what version a user is reviewing when they leave a review, unlike the App Store.

Related: #152

[SergeyBiryukov]

Replying to cais:

I'm fairly certain I have written at least one review in all my years working in and around WordPress although I have no way to easily find it since it has been more than two years since one was written

Related: #2007

[photocrati]

Replying to SergeyBiryukov:

Replying to cais:

I'm fairly certain I have written at least one review in all my years working in and around WordPress although I have no way to easily find it since it has been more than two years since one was written

Related: #2007

:grin:

[SergeyBiryukov]

Replying to cais:

I'm fairly certain I have written at least one review in all my years working in and around WordPress although I have no way to easily find it since it has been more than two years since one was written

#2007 is now live: ​https://wordpress.org/support/users/cais/reviews/

[cais]

Replying to SergeyBiryukov:

#2007 is now live: ​https://wordpress.org/support/users/cais/reviews/

Thanks for the update! Nice to see that these reviews are editable even going as far back as the few I have written although I had to look at the reviews several times to see where to edit (I know it's a great big blue banner ;) ... it just didn't register as the interface is different than the standard forums view of reviews).

[SergeyBiryukov]

Replying to samuelsidler:

One thing to consider is that we don't necessarily know what version a user is reviewing when they leave a review, unlike the App Store.

#152 has a patch to address this, anything I can do to help move it forward?

[Otto42]

One thing to consider is that we don't necessarily know what version a user is reviewing when they leave a review, unlike the App Store.

A consideration brought up is that this is highly subject to gaming via artificial version inflation. People will bump versions of apps specifically to "reset" their reviews.

[SergeyBiryukov]

Replying to Otto42:

A consideration brought up is that this is highly subject to gaming via artificial version inflation. People will bump versions of apps specifically to "reset" their reviews.

Yes, this was mentioned in comment:10 as well.

The patch on #152 doesn't reset reviews though, it just records the plugin/theme version for support topics and reviews, which would make support a bit easier, and might also come in handy for the new plugin developer dashboard that has been ​discussed at the latest Plugins Team meeting.

Having the versions data, we can decide on the best way to use it while preventing plugin developers from gaming the system. Otherwise, both #152 and this ticket can be closed as maybelater or wontfix.

[SergeyBiryukov]

#3843 was marked as a duplicate.

[dd32]

This is mostly me thinking out-loud after reviewing the raw data that powers the rating number. I apologise for the length and the random data points without any specific examples or data backing it up.

I pulled the Plugin Review data for all-time vs "Ratings from 6mths prior to the last update" and went from there.
Insights:

• Most plugins have next-to-no reviews, and limiting it to the last 6 months just causes a single 5 star (or 1 star) review to be the number behind it
• Some plugins have a lot of reviews, for example the top rated plugin (Which prompts users who actively use it to review IIRC) has 5x the reviews of the 2nd most reviewed plugin.
• Most large plugins (>1m installs) who have a decent stream of reviews coming in, generally have the same rating for all-time vs the 6mth time frame. Maybe it's 0.1 stars off, but it's rare for it to differ much.

In the data I've been working with

• 198 plugins have >10 reviews in six months and would have their rating increased by at least 0.1 stars (only 19 would gain 0.5 stars). If you limit it to >20 reviews, then it's 95 gaining 0.1, and 7 gaining more than 0.5.
• 330 plugins with >10 reviews in six months would have their rating decreased by more than 0.1 stars (93 loosing more than 0.5 stars, and 29 loosing more than 1 star). Once again, >20 reviews is 112 decreasing more than 0.1, 32 more than 0.5, and 15 loosing more than 1 star.
• 7,500 plugins would see less than a 0.1 change in their review, only 593 of those have >10 reviews
• A LOT of plugins who have never been reviewed, or have not been reviewed in the 6 months prior to their last update.

So.. options IMHO:

• Use recent reviews if it theres >20 reviews. Ie. $reviews = $six_month_num_reviews >= 20 ? $six_month_rating : $all_time_rating; but that opens up more gaming, where you can 'fix' a bad rating temporarily by spamming or drop a plugin rating by spamming (something which is seen more often than I like)
• Weighted ratings, more recent reviews are weighed heavier than old reviews
  • In my small amount of experimentation with that, it seemed to mostly match up with the averages of current with a significant impact upon performance of measuring the data
• Weighted ratings by time period, $rating = avg( $six_mth_rating, $all_time_rating ) or $rating = avg( $this_month_rating, $last_six_month_rating, $all_time_rating )
• Leave it as-is, all-time reviews. Revisit when more plugins are getting reviewed and there's more than 10 points of data for the majority of plugins.

I'm leaning towards no change here, purely because of the significant limited data points available for most plugins and the majority of plugins having no change.
The 15..29 plugins who would loose more than 1 star is a good indication that we need to start to consider changing something though, but I don't think a hard 6mth cut off (either of the last 6mths, or 6mths prior to last update) makes the rating number more accurate for most end-users. Using any of the weighted averages or multiple range averages also tends to hide these plugins as the rating drop is more like 0.3~0.5 star drop at most after applying that averaging.

edit: It was also pointed out to me, that the Apple store uses two star ratings, one for all time, and 1 for 'this version' likely to combat this exact scenario. It's unlikely to help WordPress.org though due to the low number of data points. We need more reviews in order to make the number more useful.

[dd32]

Milestone renamed

[dd32]

Milestone renamed

[mrfoxtalbot]

I suggested a different approach to editing reviews in #7064.

If we fix it, it would allow us to close this issue.

[cais]

Replying to dd32:

This is mostly me thinking out-loud after reviewing the raw data that powers the rating number. I apologise for the length and the random data points without any specific examples or data backing it up.

Although I don't feel I have any skin left in this game, this older comment had me thinking. Why not simply return "there are not enough reviews" if an arbitrary number of reviews have not been made? Then follow through with a time-delimited approach.

I wish the contributor(s) to this ticket all the best and thank them in advance for their time and efforts.

[dd32]

#5196 was marked as a duplicate.