« Dunbar Number Presentation at MeshForum 2006 | Main | Speaking about SynchroEdit at WikiWednesday »

Ratings: Who Do You Trust?

My colleague, Shannon Appelcline, has been working on a game rating system for RPGnet. This has resulted in real-world application of the principles for designing rating systems which we've previously discussed in our Collective Choice articles. Shannon's newest article, Ratings, Who Do You Trust? offers a look at weighting ratings based on reliability.

Shannon_appelcline

On the RPGnet Gaming Index we've put this all together to form a tree of weighted ratings that answer the question, who do you trust?

Here's how we measured each type of trust, and what we did about it:

  • Volume of Ratings for an Item. Introduce a bayesian weight to offset the variability of items with low-volume ratings.

  • Volume of Ratings by a User. Give each user a weight based on his volume of contribution which is applied to his ratings.

  • Depth of Content by a User. Give each rating a weight based on the depth of thought implicit in the rating which is applied to that rating.

These all get put together to create our final ratings for the Gaming Index, with each user's individual rating for an item getting multiplied by its user weight and its content weight, and then all of that averaged with the other user ratings and the bayesian weight too. The result is in no way intuitive, but users don't really need to understand the back end of a rating system. Conversely we hope it's accurate, or at least more accurate than would otherwise be true given the relatively low volume of ratings we've collected thus far.

Here are some of Shannon's earlier discussions about the design behind the new "user content" based RPGnet Gaming Index:

Related articles from this blog:

  • 2005-12: Systems for Collective Choice
  • 2005-12: Collective Choice: Rating Systems
  • 2006-01: Collective Choice: Competitive Ranking Systems
  • 2006-08: Using 5-Star Rating Systems
  • 2007-01: Experimenting with Ratings

    • Related articles from Shannon Appelcline's Trials, Triumphs & Trivialities:

  • #196: Collective Choice: Ratings, Who Do You Trust?
  • #198: Collective Choice: More Thoughts About Ratings

    • Posted on September 14, 2006 at 04:28 PM in Games, Social Software, User Interface, Web/Tech, Weblogs | Permalink

    TrackBack

    TrackBack URL for this entry:
    http://www.typepad.com/t/trackback/14990/6053392

    Listed below are links to weblogs that reference Ratings: Who Do You Trust?:

    » Reputation: just don't do it at first from Korby Parnell's Social Software Wunderkammer
    I've been catching up on Life with Alacrity (RSS), a blog by Christopher Allenthat focuses on the... [Read More]

    Tracked on Sep 29, 2006 12:13:07 PM

    Comments

    Hmm, interesting on the weighting system.

    Can you explain with more detail how the rating by depth of content is generated?

    I'm looking through the other articles, so hopefully I find the info there.

    Frank

    Posted by: magicback (Frank) at Oct 12, 2006 1:30:32 AM

    My first pass system just used a different multiplier for weight based on the type of content:

    1x for just a raw rating
    2x for a rating with a non-blank comment
    5x for a review (which goes through a different, approved content system)

    My second pass system also included "volume of ratings by user" and thus applied a variable multiplier depending on how many ratings the user has made:

    (0-2x) for a raw rating
    2x(0-2x) for a rating with comment
    5x for a review

    The 0-2x is calculated as (# of ratings by user)/50, to 2 max.

    I'm pretty sure that the core weighting system by depth is producing better results, though I haven't done any studies of that yet. The additional weighting for volume by users has definitely prevented bias by hit-and-run raters who just stop by to rate one game that they've been asked to.

    Posted by: Shannon Appelcline at Oct 16, 2006 3:39:53 PM

    Post a comment