[Home]DigestedSummaryEmpiricalResults

MeatballWiki | RecentChanges | Random Page | Indices | Categories
There is an objective, empirical way to demonstrate the current state of the DigestedSummary experiment. We can go through all the digested summaries written over the past few months and analyze then along some key dimensions. The most important to me is how frequently are summaries digested. Another key dimension is how often are summaries chained (i.e. simply appended; often with semicolons; as like this). Also, how often EditCategories are used, and which ones are used the most. Finally, we should break these down by author to see if there are biases based on expertise.

We should also be open to other social behaviours that have emerged, of course. I don't believe this list of questions is exhaustive.

We should also compare the rate that the digested summaries are changed by an editor compared to how often editors wrote a ChangeSummary? before DigestedSummary was introduced. A lower change rate would suggest the new summary format is too confusing. (Similarly, we can break this down by author.)

I think what we will discover is that by having an open-ended summary format, we will learn how people actually want to structure their change information. We can then decide how best to approach that. -- SunirShah

I wrote a simple algorithm to decide whether a summary was digested or not: if the characters of the previous digest cannot be found in the same order (though not necessarily adjacent) in the new digest, then the summary was digested, i.e. couldn't be assembled by simply concatenating edit summaries. I then manually categorized these to determine their type exactly.

All told, I analyzed 21846 revisions covering 2159 pages, taken from [1]. I divided the timespan into three sections:

I've not fully analysed the trial period numbers, will get on that soon. In the meantime, enjoy the lovely statistics :) -- ChrisPurcell

Discussion about first round of analysis

Chris, wow. This is fantastic! Great job! This is incredibly thorough, more than I even imagined.

I realized that your algorithm for deciding 'digestion' also includes people who totally replace the summary with something new and different. One way to account for that is to take a moderately sized random sample of 'digested' changes and assess (manually) on a percentage basis how many are actually digested vs. totally changed vs. whatever. We might assume that the rate remains constant over the period for expediency, even though that itself is not clear. -- SunirShah

I've analysed the "digested" summaries from June 27 onwards, categorizing them manually. 125 were spam-related, the overwhelming majority being restoring the digest after a spam attack, the rest being digest mangling during an attack. In 6, no real changes were made, only whitespace removal.

The remainder I split between mistaken changes, where people replaced the digest field when they shouldn't have, and deliberate digests, where previous summaries were rewritten or even replaced entirely based on sound editorial planning with TheAudience in mind. Note that this is an entirely subjective opinion, as one man's deliberate choice is another's mistake. I found 81 mistakes and 165 deliberate digests.

Many mistakes were of course made by newbies. However, a significant fraction were deliberately made by oldbies making up for the lack of the old non-digested summary by replacing the digest each time, usually on French CopyEdit/PageTranslation pages. Hence the subjectivity: I feel such use is inappropriate for TheAudience reading RecentChanges.

The most interesting use of digests in my opinion is as a kind of WebLog update flag: one posts only the latest update in the digest, as anyone who comes to read it will find any backlog they may have missed.

Note that some digested changes may be being misclassified as trivial, as the simple algorithm I used will not flag when a later editor adds an EditCategory to a previous editor's post unless they also change the digest text. Nevertheless, such a change could not be assembled by concatenating edit summaries, so may be considered digestion. Not sure if I'll take the time to put numbers to this hypothesis. -- ChrisPurcell

So, to summarize the conclusion here, in the most recent period, roughly 3% of the time people digested the changes and 25% of the time people simply appended their changes. That indicates a significant drop from before DigestedSummary was introduced when we had change summaries for more than 50% of edits. However, we now have a much better picture of the kinds of things that people want to do with summaries. How can we adapt the workflow to achieve a digestion rate of 10%, which I think would result in a self-sustaining social norm and community practice? -- SunirShah

I think the simplest way would be for change summaries to be focused on the particular change, but then aggregate these changes as a bulleted list rendered on RecentChanges. This bulleted list would really be a wikitext buffer that can then be digested through another interface, accessible through RecentChanges itself. -- SunirShah

You have the statistics wrong, because I presented them a bit impenetrably before. 3% were indeed digestions, but 44% of the time people added their changes.

I'm not sure where 10% comes from, seems a bit arbitrary. I think we should take a look at the distribution of changes between summary field replacement before using a pull-the-rabbit-out-of-the-hat number :) I'll get on it soon. -- ChrisPurcell

Ok, that's a lot better. It is still a statistically significant amount. Of course, I have no idea how to compute p scores, so I'm just guessing a drop from 55% to 44% is significant. The 10% number is out of my ass, but it follows from the wisdom of the TeethToTailRatio. I still wonder if most of the summarizations are not digests, do we need to digest from the edit window or from somewhere more convenient, like RecentChanges? After all, digesting a long chain of changes is orthogonal to actually editing the page text. That should increase the digestion rate, I think. And if we are simply chaining the change summaries, we can make it easier to read by putting them in a nice clean list rather than separating them with semi-colons. -- SunirShah

Perhaps the numbers you want to compare are "16%" and "15%" - the proportion of summary-less modifications before and after digests. Or "29%" and "29%" - the proportion of summary-less follow-on edits. Given that I didn't categorize any of that 55%, it's quite likely that a good proportion of it was spam. (There are no edit categories to help out on this decision, and I'm not going to manually scan 8360 pages!) That would make the figures before and after fairly comparable.

We could allow digestion from RecentChanges as well as the edit page; I don't think the numbers support removing the simple digest system from the edit page. Or we could allow digestion from the history page, as that has more pertinent information to hand, and is the place I often find myself on when I want to do some digesting. -- ChrisPurcell

Discussion

MeatballWiki | RecentChanges | Random Page | Indices | Categories
Edit text of this page | View other revisions
Search: