Last week we introduced HomeStars’ new Scoring System, the goal of which is to help homeowners make better hiring decisions. Since we made the announcement, we’ve received a number of enquiries regarding the mechanics behind the scores.

Here’s a detailed overview of how our Scoring System works:

It all starts with reviews

The foundation of our Scoring System is the reviews we collect. We rely on user-generated content from homeowners on the projects they undertake – good, bad and indifferent.

All reviews are not weighted the same.

Through market research and talking to our users, we know that people value some reviews more than others. For example, a review that was written 5 years ago by a user that has written one review has much less importance than a review from last week written by one of our gold badged Star Reviewers.

The HomeStars rating system behaves in the exactly same way – we weight the reviews by the factors that are most critical to our readers.

The importance of recency

A local consumer review study indicated that 69% of people say that a review has to be written within the last 3 months to be relevant. The rate at which recency starts to affect a rating varies by category – in the same way that the length and size of jobs do.

For example, a home inspector could have several customers in one day, whereas a general contractor might be working on a job for weeks or even months. Therefore, we would expect reviews more frequently about home inspectors.

We provide every company on HomeStars with a clear indication of their current score for recency based on the type of services they provide, in the same way we do for all factors the ratings are based on.

Reputation is calculated by our extensive moderation process

Our integrity team have been perfecting the process for verifying and publishing homeowner reviews for over 10 years. This extensive experience with both human and machine intelligence ensures the authenticity of reviews found on HomeStars.

Review moderation process

The vast majority of the reviews we receive from homeowners are honest and pass through our moderation process without any issue. However around 15% of the reviews we receive don’t meet our guidelines, for a variety of reasons.

It is not unusual for us to discover review tampering whereby companies use mechanical bots, friends and family or third party companies to post false reviews. We also encounter home professionals who bully and frighten homeowners into removing, editing or otherwise altering an honest, negative review.

In all these cases, the consequence of such behaviour is that a company’s score is lowered immediately. If the company continues tampering, negative or harmful behaviour, their score will continue to drop. If however, the company refrains from circumventing our guidelines from this point forward, the adjustments made to their rating will fade over time as their good reputation builds. We have a zero-tolerance policy on review tampering and homeowner bullying.

As a result, we have adjusted over 7,000 companies’ scores – and counting – due to suspicious behaviour identified in the moderation process. To supplement the investigation process, we also do some credit and background checking of companies, and are always looking to integrate more third-party data that will give us a more complete, real-time understanding of a contractor’s reliability and reputation.

Being responsive to homeowners helps a company’s rating

Responsiveness is important to measure as it is a better experience for homeowners if companies that are actively engaged on the platform are favoured over companies that are less responsive to customer enquiries or reviews. Responsiveness however, has less impact on a contractor’s score than other measures.

So the next time you’re searching for a home professional on HomeStars, you can be confident that the companies that are rated the highest have recent top-scoring reviews, no review tampering or suspicious behaviour, and are engaged on the platform.

One final point to remember – in no way do our paid members’ ratings differ from free members. We run a democratized system that allows any company to succeed on HomeStars, regardless of whether or not they are a paid member.

Questions about our new scoring system? Please send us an email, we’re always happy to hear your thoughts and answer any further questions.

Posted by Tom Brookes

Tom is the Director, Business Intelligence & Content at HomeStars. He is in charge of understanding the vast amount of data that Homestars collects. Tom spend most of the day drinking tea and looking at graphs.