KuroshinSubmissionQueueDiscussion

MeatballWiki | RecentChanges | Random Page | Indices | Categories

The 20 points listed on KuroshinSubmissionQueue are discussed in detail here.

: ''Wiki:DeleteMe: if anyone is worried that this page seems drastically shorter than the original, there is a simple explanation -- it appears that items 4 thru 10 were duplicated by a sloppy copy/paste, and at least one item had content from another point inserted into it. Blech. I only became aware of this once I manually numbered the 20 items, only to discover that I had 28 numbered items. Consider that a Wiki:TextSmell and a Wiki:RefactoringTechnique.] -- EricScheid

A summary of suggested changes arising from this discussion is on KuroshinSubmissionQueueSuggestions.

#1. The purpose, intent, and role of the queue are poorly defined, and as defined appear to be contradicted by implementation.

The apparent goal of the submission queue is to be able to rate submissions for the site, and allow corrections of errors or problems with the submission. Changes are not currently possible.

The ultimate goal of K5 and Scoop is: maximize content quality.

The goal of the submission queue is slightly more complex. Ideally, it's modeled on an editorial bull room, in which submission ideas can be hashed, proposed, polished, and committed to print (or webpage). This means that the submission queue has more functions than KuroshinModeration? -- there's also a QualityAssessment? function at work. An intent of the submission queue is to mirror this functionality to the greatest extent possible. I'm (KarstenSelf) somewhat hampered in this by my lack of intimate familiarity with news / publishing organizations, so step in with comments if you do.

The QA function raises some interesting issues, among them whether or not the queue should be considered a bugfix system of sorts. Point to ponder. XXX

At K5, the tasks are:

Assess submission quality.
Identify (and fix) errors and/or problems.
Identify proper section/topic for submission.

The current system barely manages to he first objective. It occasionally manages to identify errors, though they can rarely be fixed. It fails the last completely.

Though sectioning might be considered a task of the author, not the queue / editors...

Headlines and submission placement in media are determined by editorial, not copy, staff.
Current sectioning of submissions at K5 leaves much to be desired.
The possibility of soft or loose sectioning, in which a submission is associated, strongly or weakly, with multiple sections, may be more appropriate.

In order to fix the queue, the first task is figuring out what it should do. The tasks listed here are central and required.

#2. Authors need to be able to edit submissions in queue.

Very simply, problems should be fixable.

Without the option to make changes to submissions, the queue becomes a lottery, bad content is posted if it's compelling enough, the author is popular enough, or there's sufficient topical discussion that comments become the basis for moderation rather than the submission itself.

One change that is necessary to the voting process is that existing votes on a submission be cleared once changes are made. There are several possible ways of doing this, one I prefer is that a submission be capable of being "promoted to queue". Essentially, it hasn't been pulled, and it hasn't been posted, it's been resubmitted to queue. The first submission is marked as inactive (more later), the resubmission is linked back to the original.

In order to prevent abuse (and let really lame submissions die), a restriction on resubmits should exist. One option is that there would have to be some threshold of "request rewrite" type requests from moderators. Another would be to let a submission be promoted only some number, say three or four, of times. After which a full resubmission would be required, possibly counting against the author's submission quota.

Many readers (including rusty) like this idea from k5: Stories are initially placed in an editing room. While the story is in editing, users can post editorial comments (no topical comments) and the author can revise the story. Then, when the author chooses, the story moves from the editing room to the voting queue. Once in the queue, only voting is allowed with no comments.

Discussion (and some variations) on this idea:

This idea is perhaps overcomplicated, and has some nontrivial flaws as noted in the threads listed. However, any solution that meets the requirements given above will probably benefit from the ideas in the "editing room" suggestion.

#3. Authors need to be able to pull submissions in queue.

Very simply, submissions which are fatally flawed, or replaced, should be killable.

#4. As K5 grows, current posting algorithm means no submissions will post

The current posting criteria operate in such a way that with sufficiently many users, no submissions are posted. The method needs to be changed.

Modeling of the K5 post/dump criteria show that fewer and fewer submissions are posted as the user population grows. This is an outgrowth of statistics (and yes, I plan to show some convincing proof, but won't mind if someone beats me to it). While the problem could be addressed by changing the current "percent of users" thresholds to be variable rather than hard constants, better solutions exist.

Note also that this single fault is responsible for much of the brokenness of the queue as several direct and indirect consequences result.

There are three voting systems which have been proposed at K5:

The current "threshold delta" system
An "absolute threshold" system (proposed by ??? Inoshiro, thurler, or bmetzler, I think)
A "mean rating" system -- similar to the K5 comment moderation (RustyFoster, KarstenSelf)

The last is the only sane way to go.

Under "threshold delta", a certain difference between "post" and "dump" scores needs to be attained. That is, you're not asking whether or not more people wanted a submission posted than didn't, you're asking whether or not a difference between posts and dumps (a delta) was attained. The problem is that of all three methods proposed, this one doesn't guarantee that a decisive outcome will be reached. There's no guarantee that X more readers will like rather than dislike a submission, no matter how far out you take the voting. They could simply split it down the middle. What ends up happening is that submissions are dumped with several hundred people indicating that they'd like to see it posted, and with involved discussions occurring under the topic, all of which are lost. This is a bug. This option tends to have more uncertain outcomes (not hitting a threshold) than certain. Assuming a normal distribution (it's not, but the conclusions are still valid), what happens is that more and more of your outcomes are uncertain the larger the population grows (I'm going to fudge and say this is the central limit theorem, but I think I'm wrong). In other words, the more successful K5 is, the worse it gets.

An absolute threshold at least solves this problem. Here the system is that the first threshold to be met, post or dump, is the one that is effective. Effectively, you're setting up a race between those who want to see a submission posted, and those who want to see it dumped, and the first group to pull in a big enough team, wins. The problem is that you're losing some useful data, and probably skewing results a bit -- a submission that is just one vote away from "post" or "dump" is disposed as if it were given all "post" or all "dump" votes.

The "mean rating" system provides several benefits.

It is the only system which provides a basis for ordinal comparison of submissions.
It is the only system which readily facilitates rescaling criteria as site volumes change (or among sites of different volumes) in a predictable way.
It is the only system which guarantees both an outcome and a fixed participation set.

Mean rating works by having each user indicate a preference on a submission. The current moderation system uses a scale of 1-5, with a few (1% of total) users being "trusted", and capable of rating a 0 for spam, trolls, and similar dreck. After a sufficient number of moderations are collected, the mean is computed and a disposition of the submission determined. My suggested threshold would be:

High (say, 3-10 articles per site per day): Front page
Medium (say, 3-10 articles per section per day): Section
Low (most of remainder): Diary of submitting author.
Spam (submissions with <1 moderation): dump as spam.

How many moderations to be considered (aka "quorum") is a good question. I suspect the answer is somewhere between 30 and 60, possibly less for some sections (e.g.: news). I'd like to run some Monte Carlo simulations to see what sort of variance might be expected given historical patterns (I've got queue data for about 20 submissions a week or so back). Statistics does a pretty good job of saying you don't need a really big sample. I don't think anything over 100 moderations is necessary at all.

: Others may not agree with this assessment :-) With a quorum of only 60 to 100 moderations the operation of the queue may become "too fast." 100 moderations can be achieved in a remarkably short period of time (<2 hours) given standard voting behaviour. This allows rapid posting of time-sensitive content and as such is an improvement over the current queue. It can have negative effects in that breaking stories with incorrect preliminary information may be posted. Since incorrect preliminary information is a hazard of even established news organizations the measure how detrimental this may be is uncertain.

: The short "hang time" of a submission which would result from a small quorum may have other unintended results. As the total readership grows and becomes more diverse and the size of the quorum required to get a story posted becomes smaller the likelihood of a full quorum of persons unqualified to critically judge the content of a submission becomes greater. With a greater hang time the likelihood that a clueful moderator will arrive and point out the flaws in factuality, reasoning, etc. increases. (I do not point out the solution of socially engineering unqualified moderators to abstain from voting as this is unlikely to work.)

: Another issue with such a low quorum is the possibility of non-standard voting behaviour (trolling). With the reasonably-organized large groups of trolls that are in vogue now this could become a major problem. All that would be required with a low quorum number is for a sufficiently large group of trolls to time-coordinate an attack on the queue. As the size of the quorum increases the likelihood of a group of trolls large enough to attack the queue decreases.
: -- MarkDuPrey?

In addition (or in lieu of) moderating a submission, a moderator could indicate a categorical rating (see below). Section and topic placements should also be specifiable.

A submission's current status should be available to all registered users viewing the queue at any given time. This would include the current mean rating, the number of ratings, the standard deviation, and additional characteristics (see below -- categorical ratings, sectioning), with a link to explode detail similar to the current moderation detail feature of ScoopEngine.

#5. Submissions hang too long in queue.

This is both frustrating and unnecessary.

It's frustrating because basic tenet of a decisionmaking process is that it should be efficient. Make a decision when you have sufficient data to do so. So so sooner rather than later. Where iterated development is being practiced (Scoop sites being one example, FreeSoftware development being another), more rapidly interated cycling wins over slower iteration.

It's unnecessary because the additional data provided beyond the first 30-60 votes (this is a SWAG, but should be supportable) don't provide any additional data on the problem. The ratio of "post" to "dump" moderations isn't generally going to change appreciably over time.

Hang time is largely a consequence of the current moderation system, it will be fixed with adoption of mean rating and a sufficiently low quorum. Focus here is on the problems it causes.

Submissions hanging in queue is frustrating to users who've already moderated. One characteristic of a well-designed system is to have rapid response. K5 is anything but.
Increasing hang time means an increasing likelihood a submission will be dropped. Maximum hang time is for a submission with a dead-even split, which won't cross either the drop or post threshold. But a submission that's not posted is dropped.
With increasing hang time comes increasing topical discussion. This means that there's more personal investment from users the longer a submission's in queue. But the submission's more likely to be dropped (see above). This leads to user frustration. Bad system design.
Increasing hang time means that there is less fresh content on the front page and sections for registered users. We've already seen the topic in queue.
Increasing hang time means more users have to moderate each submission, and each user is moderating more submissions. Part of the magic of CollaborativeFiltering is letting someone else do the work. Instead, each user is moderating each submission, then having to deal with it again. Bad system design.
You frustrate your better users. We're frustrated that submissions we've moderated are still in queue. That our topical comments are lost. That there's little fresh content. That we have to moderate everything (and a fair bit is dreck). Result: talent leaves for MeatballWiki ;-)

The solution: reduce hang time by changing the voting system. The problems will go away or at least be radically diminished. If you don't resolve the problem, you've got a vicious spiral.

#6. Queued submissions generate too much topical discussion.

This obscures editorial content, skews moderation biases, and frustrates users whose posts "disappear".

Also, as seen above, topical discussion (in queue -- mind, discussion in posted submissions is good) is a symptom of problems. Of course, allowing topical discussion in queue was also a palliative treatment of symptoms as well.

Solution: by improving the voting system and reducing hang time, there's no need for topical comments in queue. Remove the "feature". Editorial comments are allowed, but aren't visible (by default) nor may they be replied to, once a submission's been posted.

#7. Queued submissions generate too little editorial discussion.

Observation. Possibly not a bug, but I think it's a problem.

Possible causes: too much topical discussion, hang-time's too long (or too short?), there's no way to fix mistakes, moderation doesn't work.

One major cause: Editorial comments seldom have any effect. Authors seldom post revised versions of stories, especially if the original version will be accepted despite its flaws. Because of some of the other points listed above (authors cannot revise or pull submissions; no categorical feedback; etc.), the site does little to encourage revision. Therefore, criticism is generally not useful in any practical sense.

#8. Categorical feedback on submissions should exist.

Checkbox items for typical requests, problems.

The problem with this is coming up with a good list. A good list is both sufficient and short. Ideally this is tied to a little bar chart like current poll responses indicating article status for a quick scan.

Post as-is: Submission is suitable for posting.
Fix spelling, grammar, mechanics, tags: Fix errors, request rewrite.
Fix factual, flow, or bias: More fundamental errors, fix, request rewrite.
Not suitable for posting: Submission is fundamentally flawed, duplicate, etc. Can be resectioned to author's diary.
Spam / troll.

"Request rewrite" means that the author should fix the problems, then repost (through the edit mechanism) a rewritten version. Moderation and comments start fresh.

#9. Feedback in the form of submissions posted / not posted should be provided.

Worse than its failings (which are highly visible) is the fact that the queue's successes are invisible. The only time people are consciously aware of the sub queue is when it's broken. At some level, a visible record of what's being done on the site should be visible. Summary statistics would be useful. Better yet, post failed submissions (that don't hit the spam threshold) to the author's diary.

#10. Sectioning of submissions is erratic and often irrational.

This reflects problems with both sections and the submission process.

There seem to be a number of authors who don't have a clear idea of how sections should be used. The available sections and topics at K5 should also be rethought -- there's a bit of site management that has to happen here. The News/MLP distinction is a big one. I'd also like to see a place for the high school / PoliSci? crowd to go where I don't necessarily have to see or hear them....

Preference: When moderating submissions, it's possible to select the section(s) and/or topic(s) a submission should be posted to. Multiple-choice checkboxes type thing. Another poll-type graph showing current status. It is possible for a submission to get sectioned to more than one spot, possibly with different rankings in each, according to emphasis given in the moderation queue, which would effect final placement. This also suggests that a metric be used to determine placement of submissions within section based on a compound index, to include: submission moderation, article publication date, article reads, comment activity (active comments on the article and an aggregate of their rating(s)), XXX.

Another possibility is to have moderator-submitted headlines if the author-supplied one is considered poor or inappropriate. This also mirrors real-life publications in which copy and headlines are provided by different functions.

#11. Voting preferences vary greatly by section.

Users of K5 have different interest. Users may have little or no interest in reading, or moderating, submissions in different sections of the site.

See above on sections.

I've little or no interest in PoliSci?. I don't need to read the various blather forums, I don't need to moderate what goes into them. One possibility would be to have submissions directed to specific sections. People reading these sections would see that there were new submissions for moderation. Possible problem: retaliatory moderation by users of one section against another. Possible extension would be to have some form of trust metric or rating of raters within sections. Hmmm.... XXX

At least one scoop developer has attempted a "customizable front page" feature, which would ideally let users change the front page to include or exclude all stories from a given section or topic. This could allow readers with different interests to coexist with less interference. See GlobalResource for discussion.

#12. The best submissions are frequently dumped.

Current voting algorithm only allows in very highly rated submissions. Debatable submissions languish, and eventually, die, usually taking much merited discussion down with them.

This is another consequence of the current voting scheme. Submissions can get middlin' scores two ways: by being middlin' (which everyone agrees on), or by being Great! No! It's Shit!....in which case a conflict of opinion results in a middlin' score. But generates some strong discussion. This is the content that a site like K5 lives and dies by. Right now, it's falling by the wayside.

#13. Moderation doesn't get used meaningfully.

Other than a post/dump decision, a submission's queue performance matters little.

Just yet another gripe about the current system failing to provide good data. With a moderation score, things like (shock, amazement) placement on a page, could be determined in part by a submission's moderation. Note also that moderation of submissions can continue even after they've left the queue.

#14. Diary entries are effectively ignored by the site.

A method for promoting good diary material should exist.

Moderation provides a ready linkage for identifying interesting diary material. This could be promoted through a "best of diaries" list, or even by direct promotion of diary entries to front page or submission queue (with diarist's approval).

A simple step in this direction: Allow users to "subscribe" to a diary, placing its entries on their custom front page. To be very useful, this would need to be part of a larger customizable front page project which also allowed "subscription" to story authors, topics and/or sections.

#15. Queue performance should impact authors.

An author's queue submissions should measure against mojo.

Moderation again provides a mechanism for doing so. How to do this is another question. Think....XXX.

#16. Queue access should reflect past history.

An author's access in queue should be partially driven by prior status and history, with possible limits on submissions in a time period.

Personal opinion, but there are some people whose names are appearing in the submission queue far too often. How to throttle this is an open question, but I'll posit it should be done.

#17. A mechanism for author's comments is required.

There's a place for context regarding a post which are neither comment nor submission.

The typical example is a submission that's been reposted from some other location. The author wants to add some information that's of significance while the submission's in queue. There's currently no place to do this. I'd like to see an "additional notes" sort of section added where this can be done.

The online documentation suggests editorial comments for this purpose. They meet the main need (visible only in queue). The drawback is that they are not highly visible, and can be lost in

#18. Better moderation directions.

The moderation page itself should be made more explicit rather than less. With directions on what's happening and why. Fewer moderations of higher quality. That's the goal.

#19. The Queue is not a scheduling system.

The submission queue has been used as a system for scheduling submissions. It's not.

There is a practical use for scheduling content at a ScoopEngine site, both at a large one such as K5, and at a smaller internal site used as a ContentManagementSystem?. Using the submission queue to provide this functionality is broken in several ways. The functionality should be provided separately.

Why is this behavior broken?

This isn't the primary intent of the submission queue.
This exploits a misfeature of the submission queue which should be (will be) fixed.
The submission queue doesn't solve this problem cleanly -- scheduling is unpredictable and lumpy.

Suggested fix: Provide a separate scheduling capability to the submission queue. Submissions can be aimed at specific dates or times. There are two aspects of this: site management (in which a site determines to some extent or another a calendar or schedule in which submissions are accepted) and author scheduling (in which time-sensitive material might have a "not to appear before" or "drop dead by" date -- a publication window. A particular submission might specify a minimum, a maximum, both, or no restrictions.

For a site such as K5, looking out more than a week or so may not appeal, though coming up with an editorial calendar and topics to suit could be of use. As submissions are entered into the queue, options of when the submission could appear (open slots) would be indicated. For slots which aren't conclusively filled (there are queued submissions which haven't been finalized), indicate a potential open, but also indicate the first guaranteed open spot. There is a problem in that the order in which submissions come out of queue isn't guaranteed, but in general a sufficient pipeline will mean that a later-submitted submission can pop in to fill an earlier timeslot if it should open up.

There's also the question of how to manage moderation of submissions with a "do not post before" date. These might include press releases or other time-sensitive content. Typically in the real world these are "embargoed" -- released to the press but not to be published or publicized before a specified time -- sometimes very precisely specified, as in down to the minute. Whether or not ScoopEngine should support embargoed drops which don't hit the queue itself until some specific time is an open question. I'd leave it as a lower-priority feature but open the question.

#20. Author profile should be expanded.

A better form of providing information, background, and a profile on authors should be provided.

This spills over into other site features, particularly search, which is also broken.

An author bio should be available. This would be part of a user's information, would be a limited-length text field, and could contain whatever the author wanted to supply. Could possibly be the same as the current user bio or different, but I'm leaning toward different.

For author information, a summary of submissions, disposition of submissions (posted, pulled, mean rating, standard deviation) should be provided. The number of comments and diary entries should also be provided rather than just links to "articles by, diaries by, comments by...".

For search in general, "author" should be a separate criterion from keywords. I don't want to search for "foo bar" or "written by JoeRandomLuser?", I want to search for "foo bar" AND "written by JoeRandomLuser?". Date window(s) should also be provides (articles posted before/since). Searching by title, intro, body, or any. See Google's advanced search or the old Deja search interface.

CategoryKuroshin