RatingAsContent

MeatballWiki | RecentChanges | Random Page | Indices | Categories

There is a continuum between ratings (or votes or whatever) and normal content (such as written text). There is little notional difference between adding a comment saying "I agree!" and giving the original a high rating for accuracy. Or a comment saying, "That's insightful!" and a rating for insight.

There are some differences, mainly that rating/voting can be understood by machines. They can construct summaries easily. 100 positive votes can be collapsed into a single number without much loss, whereas 100 "Me too!" comments are pretty hard for anybody to grasp.

If we want to improve the signal to noise ratio (and we do), clearly we want to encourage rating and voting, and discourage commenting. Instead of writing new comments, people should find an existing comment which expresses their views, and vote it up. This results in less effort for them in expressing their views, and less effort for TheAudience in having to wade through unoriginal trash. Writing a new comment should be a last resort.

The downside is that ratings and votes can be generated without much thought too. It's much easier to stuff a ballot box with 100 positive votes than to write 100 original comments. This means we cannot use "number of votes" as a metric to verify that an online identity is a human with a high investment in debate. Sadly, this is an argument for saying that newcomers can comment but not vote.

This is one of my basic criticisms of the SlashDot approach to moderation. In SlashDot, moderation is seen as vastly different to normal commenting. You cannot rate and comment in the same discussion. Moderators are supposed to be impartial, commentors are partisan. This is all nonsense if ratings and comments are both content.

SlashDot also makes it easier to add new comments than to rate. Only a relatively few, privileged group of people can moderate. The effect of this is to reduce the signal to noise ratio; the only way for many people to express themselves is by writing a comment, no matter how unoriginal their ideas may be.

It seems natural from the above that comments and votes be unified in some way. Let's say the "add comment" page has both a text edit window, and a drop-down combo with a list of cliches which the machine can use to summarise the comment. If you fill in the text and leave a default cliche, it is treated like a comment. If you select a cliche and leave the text blank, it is a rating. You can also give a rating, like "erroneous", and include an explanation of the error in the text. Readers would have the option of viewing comments in summary form, ie just a list of cliches with their respective counts, or of "drilling down" to see the comments and authors.

Naturally a single user would be allowed to make several comments to the same article. Hence they could also give multiple ratings. "Original", "Bad grammar", "My experience differs" might be possible cliches along different axes, and the same author might rate the same article with all of them.

A major difficulty with this approach will be picking the list of cliches. We want enough structure that we coalesce similar opinions, without over-simplifying or overly restricting free thought. Will any fixed list be adequate, or must users be allowed to add their own cliches on the fly? If they can add their own, can we discourage them from adding too many unique ones?

Perhaps it will help if we initially populate the drop-down with the choices of previous commentors for this specific message. That way an original choice by one user may be picked up by others. The author of a message might be allowed to suggest initial cliche-options too, according to the type of his article and his personal ideology. For example, he might suggest the Wiki:SixThinkingHats system for a Wiki:BrainStorm, or a set of High/Medium/Low-priority options for a "new feature" proposal.

That example suggests we sometimes want groups of cliches, with options mutually exclusive within the group but independant of other groups. We probably want numeric options too, that the system can process.

Hopefully one or more systems will emerge and can be put into some reuseable form. I think some of the issues are similar to those of LinkPatterns?, and similar insights may apply. Eg arbitrary restrictions, such as every cliche being expressed as two words plus a number, might help channel users along useful lines.

It seems natural from the above that votes should themselves be possible targets for other votes and comments. This raises the possibility of a calculus of votes. For example, if Alice gives a negative vote and Bob gives a negative vote on that, should we consider Bob as voting positively on the original?

: If there are only positive and negative votes, then you might be able to infer that Bob would vote positively. However, if there is a neutral vote option (or a way to abstain from voting), it is more difficult to infer what Bob thinks of the original. For instance, suppose Fred wrote the original content, and it is well-known that Alice strongly dislikes Fred. Bob might read the original, and before seeing the ratings think it is a solid "neutral" vote--not worthy of a negative or positive vote. Bob then sees Alice's negative rating, and thinks it is undeserved (perhaps Alice used a cliche like "FlameBait" that doesn't fit Fred's content). Bob then might mark Alice's vote as negative simply to counter the undeserved rating, but without endorsing Fred's content. (Similar events often occur on KuroShin with people giving high ratings to counteract undeserved 0-ratings (which are supposed to be used only for spam-like content).)

: RatingGroups could provide an alternative to "meta"-voting.

CliffordAdams rated this page "+1, insightful" :-)

Rating system changes vs. deep structural changes.

Rating conflicts come from conflicting values, as Cliff points out. Some people prefer better presented material, some people prefer better detailed information, some people prefer more entertaining prose. A customizable rating system can go quite a distance in solving this problem.

However, there are values enforced by the structure of the medium that cannot be fixed through ratings. That is, each online community comes with its own purpose and is structured to accomodate that. Mailing lists and newsgroups are best for short-term, "just in time" discussion. Wikis, or at least MeatballWiki, serves best as a repository of collected experience.

A WebLog has two major incarnations, or at least flavours: news oriented and discussion oriented. Some news sites like [ZDNet (ZDNN)] allow comments on actual news articles. Other WebLogs, like KuroShin, offer mere PromptingStatement?s as the article body. Other WebLogs, like SlashDot, fall inbetween.

In either case, the comments on a WebLog are generally discoursive. But consider what happens when comments are in reply to a news article. Taking SlashDot as [an example], many of the top rated comments add to the information provided in the article (sometimes correcting it). Sure opinions surface, but they aren't wild opinions. On the [other hand], the top rated comments to an opinion piece are typically wildly rhetorical (passionate).

In the former case, the goal is to emphasize objective information. However, often the granularity of that comments is too coarse not to bring up rhetoric as well. For instance, "No, you're wrong. The actual facts are foo, bar, and baz. I think your opinion shows how the world is a sorry place to live," really adds the facts foo, bar and baz. Other material can probably disappear from emphasis.

In the latter case, the discussion follows the SoapBox metaphor pretty vividly. Essentially, those with the most interesting things to say get rated higher. This is because entertaining comments are more valuable. After all, a WebLog is something to pass the time, not a committee to design network protocols.

I think that a WebLog rating system predominantly emphasizes the latter mode of rating. I think that's often the right thing to do, as well. A WebLog, like KuroShin, focuses on interesting discussion, not informative discussion just by its dominant structure (PromptingStatement? + commentary). Consequently, material that will entertain its readership is more valuable than material that is merely informative.

However, if the goal is to have a news site that is reader peer reviewed (as was the original goal of user comments), the structure of the site needs to change to reflect that goal. One idea would be to iteratively edit the main article based on discussion below until it was better. Another would be along the lines of ArchivingNews.

Whatever; the point is that the rating system alone isn't enough to satisfy conflict values. Ultimately, the structure will have to be changed for those who dislike the PromptingStatement? + commentary style. Or even to move past the old media influence into the more forgiving new media environment. -- SunirShah

Agreed. However, see WikiLog for some discussion of the notion of "tunable deep structure." Eg:

: Postulated, then, is a system which can act exactly like either a Wiki or a WebLog, depending on how you use it. Having established those endpoints, what gets really interesting is what happens in the middle.

Although Weblogs current using rating and Wikis don't, I think it makes a lot of sense to incorporate rating/voting into a Wiki-like system. One could have a side-by-side comparison of different versions of a page, with people voting for the one they preferred. There have been several attempts to use voting in Ward's Wiki. -- DaveHarris

By the way, the notion of customisable ratings systems is a red herring for this page. My point is that rating=content and that signal to noise can be improved by encouraging rating and discouraging commenting.

: I think that customisable rating will be much more important if rating begins to replace commenting. For example, consider the ratings on [my Kuro5hin comments]. The comment I spent the most time on (and which I think is most likely to be useful to another KuroShin reader) is [this comment] about FirstAmendment issues. As of January 23, 2001, it is rated 3.25, which is my lowest-rated comment. One person thought it was great, and gave it a 5. Another rated it as an above-average 4. A third rater thought it was average quality and gave it a 3 rating. One person gave the comment a 1 rating.

: I fully support the ability of the last rater to give a 1 rating to my comment, even though I consider that rating to be inaccurate. (Personally, I thought the 5 rating was also too high.) Unfortunately, that 1 rating meant that the people who think that my comment should have a 4 or 5 rating (at least two of them) were overruled by the actions of 1 person. Several people on KuroShin are very unhappy about this style of 1-rating, and some of them "correct" such ratings by giving artificially high ratings to force the average closer to the middle.

: On the other hand, suppose that my comment really was bad, and deserved a low rating. The rating of 1 was not at all obvious from the final score. In a sense, the content of that 1-rating (the opinion that the comment was very low-quality) was removed. Indeed, many people would assume upon seeing a score like 3.25/4 that 3 people rated it a 3, and that 1 person rated it a 4. Instead, the real ratings showed a disagreement all over the quality scale.

: Customized rating systems could allow multiple conflicting rating groups to coexist without conflicting. Those who agree with [ZanThrax] (who gave the 5 rating) could work together without being hindered by users like [dj@] (who gave the 1 rating). --CliffordAdams

: After writing the above, I thought I'd take a look at the comments written by ZanThrax? and dr@. I was expecting dr@ to be one of the trolls/flamers/rebels of the K5 community. Instead, I was surprised to see several well-written and indeed above-average comments, like [this comment about the new visible ratings]. Perhaps the 1 rating was "tactical", used to counteract an overly-enthusiastic 5 rating. Maybe all of this is an overreaction--maybe dj@'s finger slipped on the mouse. :-) --CliffordAdams

: [I might move the paragraphs above to KuroshinRatingIssues or even a new page like CustomRating?. Any opinions?]

There is a real danger of trying to be too general (and indeed this may be part of why I've not implemented anything worthwhile myself yet). On the other hand, in the spirit of AlternateHardAndSoftLayers? we may need to define a language in which ideal systems can be expressed before we can properly think about what the ideal systems are. Which bits can reasonably be left "soft" and configurable, and which are "hard" primitive concepts.

: Another approach is to create a sample near-ideal system in hard/inflexible code, and then to see which parts of the system you want to make flexible. Sometimes its easier to just write/rewrite/rewrite-again to make the right thing rather than trying to make a perfect system and avoid the rewrites. Mistakes can be powerful learning tools.

I've written a similar idea up as part of my FuzzyCommunity theorising. --PaulMillar

[CategoryRatingSystem] [CategoryConflict] [CategoryUncommonWikiTechnology]

See wiki with survey feature at http://www.hanoi.dynu.com/~hanh/phpwiki-1.3.3/index.php

The survey can use as rating the content also, since the least active survey + content will be automatically deleted.

RatingAsContent

Discussion