WordPress.org

Making WordPress.org

Opened 2 years ago

Last modified 19 months ago

#2686 new enhancement

Plugin Search: Search should also take into account the number of ratings

Reported by: bfintal Owned by:
Milestone: Plugin Directory v3 - Future Priority: normal
Component: Plugin Directory Keywords: needs-patch
Cc:

Description

The overall rating is used during plugin searches while the number of ratings isn't being taken into account.

https://meta.trac.wordpress.org/browser/sites/trunk/wordpress.org/public_html/wp-content/plugins/plugin-directory/libs/site-search/jetpack-search.php#L1008

Because of this, a plugin that only one or two 5 star ratings would ranker higher (assuming all other things are similar) than a plugin that has a collective 4.5 stars from 50 ratings.

I would like to suggest that the number of ratings be used alongside the overall rating, where a higher number of ratings would have more impact.

There should be a threshold though, so as not to always favor the more established plugins. Perhaps if 20 ratings have been reached, then the number of ratings should no longer have any impact. We can probably safely assume that the rating has evened out after 20 ratings. For example, if a 4.5 star plugin has 200 ratings, then it shouldn't have much of an advantage over a 4.4 star plugin that has 20 ratings.

Change History (13)

#1 @gibrown
2 years ago

Do you have any examples of searches and poor results where you think this will make a significant difference?

Otherwise I think this will just encourage plugins to ask for more ratings.

It came up earlier to just look at total number of one star ratings rather than trying to boost. Or on whether the ratings got replied to. Whether something is getting replied to seems like a good metric really. It indicates that the plugin author is paying attention, and is trying to give real support to real users. It is also something that is under the plugin author's control and doesn't reward folks for nudging lots of users to just give them reviews with no content in them.

#2 @bfintal
2 years ago

Hey @gibrown, thanks for replying and your work on this area :)

I think this mostly happens in some-what long tail searches where there are fewer big players. The advantage also isn't enormous, but it is present.

My suggestion isn't so that plugin creators would resort to ask for more ratings, I'm not in favor of that. But instead, it's so that a having a very few number of ratings - maybe by the author themselves, maybe a friend or a handful of testers or early adopters - won't result right away in an advantage. That's why I mentioned a cut off like 20 ratings (I just pulled that number from the top of my head). My logic is that if a plugin has lower than 20 ratings, the overall rating should have a lower impact than others with 20 or more ratings. But it doesn't mean that having 300 ratings should always trump a plugin that has 50 ratings. After 20 ratings, then we can assume that the overall rating has normalized to a stable value.

For example:

(I've got nothing against these plugins, I'm just using them as examples)

  1. "Responsive Menu": https://wordpress.org/plugins/search/responsive+menu/

Awesome Responsive Menu is higher than WP Mobile Menu. It may have something to do with the name of the plugin not matching the word "Responsive" though. But I think the amount of ratings (6 vs 45) should make WP Mobile Menu appear higher.

  1. "Page Builder": https://wordpress.org/plugins/search/page+builder/

Full disclosure, my plugin there is Page Builder Sandwich so I won't dive into details in this one :)

  1. "Under Construction": https://wordpress.org/plugins/search/under+construction/

Eazy Under Construction with one 5-star rating outranks a lot of similar ones in page 2 who have 30+ ratings. Although the naming may have something to do with this, maybe the rating bumped it higher.

As for replying to ratings, I'm not sure about this. I imagine creators replying generic stuff like "Thanks for the rating, we'll do our best to blah blah blah" to 5-star ratings and actual replies would go to 4-star ratings and lower. Replying to support threads might be a better way to go, although this is taken into account already.

Last edited 2 years ago by bfintal (previous) (diff)

#3 @gibrown
2 years ago

Great examples, thanks. I'll take a look at these in the context of other changes. It's all pretty small adjustments which I think are going to be really hard to hand tune. But I'd bet that we have a lot of cases where some minor adjustments would help.

#4 @Otto42
2 years ago

Note: we encourage plugin authors to "ask" for reviews. We only tell them not to pay for them in some way, even by offering free stuff in exchange. We actively remove reviews that are paid for when we find out about them.

#5 @bfintal
2 years ago

@gibrown I think it's pretty much about the small adjustments and refinements now since the majority of the search query put out great results in the most part. I would put out a patch but I cannot find anywhere the indices being used, I found a gist that you posted 8-something months ago, but that seems outdated already. I think those are private and in wpcom.

@Otto42 Let me correct myself, what you said is what I meant. I've only added the reasoning that a small amount of reviews won't be a good enough sample size or indicator of an overall rating :)

#6 follow-up: @gibrown
2 years ago

@bfintal the index can get queried directly via a .com api. Here's some code I use to run bulk sets of queries against the index for testing the latest search alg: https://gist.github.com/gibrown/9b54444cb23fb61f4e6513a45163e98c

If you have adjustments to the query you can also just try post them on here and then I'll try them against the 3000 queries I've been running against during my testing to see how they work on a wider scale.

#7 in reply to: ↑ 6 @bfintal
2 years ago

@gibrown

I'm not sure if the number of ratings are indexed, so I can't test with the scripts you gave. But here's what I have in mind, this is to replace the last field_value_factor for the rating field:

{
  "filter": {
    "range": {
      "ratings_num": {
        "gte": 20
      }
    }
  },
  "field_value_factor": {
	"field": "rating",
	"factor": 0.25,
	"modifier": "sqrt",
	"missing": 2.5
  }
}

The 20 there is an arbitrary value and can be replaced with anything which can be deemed as a minimum sample size for an overall rating, it can be 10 or 8. This is to prevent plugins with only one or a handful ratings from being favored. I can't find the field for the number of ratings, so I placed it as ratings_num above to illustrate what I have in mind.

#8 @gibrown
2 years ago

Ah ya an example doc would probably help.

Here is Jetpack: https://gist.github.com/gibrown/f236af8a1e5365f7c8493ad601d25b74

And here are the mappings for the index: https://gist.github.com/gibrown/49262d39edb5ce7936fc001990ece560

At a quick glance, my main concern is that having a fixed number as a filter creates a step function and I think that in general we'd do better to have smooth functions in how we rank things. So I'd rather have a decay for small numbers of ratings.

#9 @bfintal
2 years ago

@gibrown

Good point and thanks for the links. Here's a smoother one that adds a multiplier 0.75 to 1.0 when it's below 20. This one actually works with the script you gave. The values for the range filter 20 and the offset and decay can be further fine tuned. The decay can be increased to a higher value to lessen the impact of this filter.

{
	"field_value_factor": {
		"field": "rating",
		"factor": 0.25,
		"modifier": "sqrt",
		"missing": 2.5
	}
},
{
	"filter": {
		"range": {
			"num_ratings": {
				"lte": 20
			}
		}
	},
	"gauss": {
		"num_ratings": {
			"origin": 20,
			"offset": 0,
			"scale": 20,
			"decay": 0.75
		}
	}
}

#10 @Ipstenu
2 years ago

  • Summary changed from Plugin search should also take into account the number of ratings to Plugin Search: Search should also take into account the number of ratings

#11 follow-up: @gibrown
2 years ago

Ah ha!

Wilson score confidence interval for a Bernoulli parameter from http://www.evanmiller.org/how-not-to-sort-by-average-rating.html

I think I will try putting something like this into the index. I need to figure how to make it work on a 5 star scale, but this addresses a lot of my concerns with hard coding any limits. Noting here mostly as a reminder.

#12 in reply to: ↑ 11 @bfintal
2 years ago

@gibrown Looks promising!

#13 @tellyworth
19 months ago

  • Keywords needs-patch added
Note: See TracTickets for help on using tickets.