We had a good question come in a few days ago, and I thought one that we’d like to answer for everyone, rather than just respond directly to the company.
I will paraphrase the question as follows:
I can’t understand why “Company Name” got the Best of 2009 award in “Category X”?
If you read their reviews, most of them are about work performed in a different category?
To top if off, the only review about their winning category isn’t actually good.
To me there is no common sense in this…
I think that’s a great question, and a bit of a tough one to answer. I’ll walk you through how we determine our awards, which is almost completely mathematical. As with great mathematical theories, the reality doesn’t always fare perfectly with the math.
The “Best of” awards are determined by looking only at reviews for that particular year. While prior years’ reviews are used in the overall average and ranking, only the reviews for that year are used for the awards calculation. How we determine the ranking on the site is how we determine the winner. We have an algorithm which combines both the number of reviews, and the ratings themselves into a single number. That is number we are using to determine winners.
There are, however, two complexities to the situation, geography and categories.
Geography, so far in the growth of the site, has been a relatively simple. We use the original address and service area of the company to determine which geography the company wins in. A company like Tender Touch Moving, will compete with moving companies in Toronto, as it’s their home base, even though they are listed in Calgary, Edmonton and Vancouver. Tender Touch, using our current rules, isn’t eligible to win in their other service area cities.
Categories, which is the concern of this user, is more complex. Our premium listing
allows companies to list themselves in multiple categories for more visibility and because they often do many things. It is one of the privileges of paying. (and, yes, we need to makes some money to keep making the site better). When we’re determining winners, we look at all each category individually one-by-one, and sort everyone listed in that category by the # in the algorithm. The leading company in that category is then chosen the winner. One of the rules we put in place was that a company can only win in a single category. So a winner who is listed in multiple categories, once they have won in that category, becomes ineligible for further categories. So as we go down categories we determine winners in each based on this method. It’s as best we can make it to be determined by pure numbers.
In this case, the company in question was listed in multiple categories, and they do not have a lot of reviews in the category they won in. That’s where good theory and reality collide. They won on the math, as we do not ‘choose’ winners. Winners are chosen by our readers, based on their experiences with the companies. The particular company in question, although they may not to a lot of the jobs in the category they won in, they do want to be listed in that category, thus they are eligible.
In order to keep the Best of awards as customer chosen as possible, we need to use the math, rather than start reading each review and deciding which categories it falls in. Even in that case, many reviews fall in multiple categories. We have a lot of reviews to look at – in the many 10s of thousands each year, so it would be next to impossible (with our resources) to analyze each one. Even if we did, many reviews fall in multiple categories. Many reviews talk about a range of projects a particular contractor did, in many categories.
Yes, we know it’s not perfect! Probably not even close to perfect – and to a large extent, our questioner is right on – it lacks a bit of common sense. But it’s the closest we can come right now.
Should we do it differently next year? Tell us how in the comments below.