TransClusion

MeatballWiki | RecentChanges | Random Page | Indices | Categories

Another great idea from TedNelson and the XanaduProject.

A transclusion is the reuse in whole or in part of another node in one node's rendering. That is, one node including another node when it is being displayed. A transclusion is different from pure copying, however, in that only a reference to the foreign material is stored. The reader's client software is expected to fetch the foreign material and place it inline with the main material.

For instance, if I wrote a report on wheat farming and I was talking about the relative price of wheat around the world, I'd want to place a chart of those prices on my site. Instead of creating the chart myself, I could just transclude the chart from the World Trade Organization directly in my report.

More realistically, on the WorldWideWeb, one can transclude images from external servers with the IMG tag. e.g.

is actually hosted on http://sunir.org, not on the MeatballWiki server.

: Transclusion is therefore the usual way of including images on someone else's Wiki site, as you rarely have the facility to upload your own images. Instead you put them on your own site and transclude them into the wiki page

And more recently, entire pages with the FRAME and (even better) IFRAME tags.

Copyright

Transclusion is fundamental to TransCopyright. See also InvoluntaryTransclusion. However, failing TransCopyright, all one needs is the RightToInclude.

If CopyrightingLinks is possible, what would full-blown TransClusion do to copyright? It's not just a link, but inline material so it looks like it's part of the rendered work. That's very close to just copying something. TransCopyright is one thing, but that's not likely to become institutionalized before the technology really gets going.

Implementations

TransClusion has been implemented using UseMod:WikiPatches/RawWikiInclusion. Whether this kind of system is usable or desirable remains to be seen. -- AlexSchroeder

There is a very rudimentary implementation of TransClusion in PurpleWiki. -- ChrisDent

MoinMoin provides the [content] action "for transclusion into static web pages, this action emits the pure page content, without any <html>, <head>, or <body> tags."

[Compendium] also supports TransClusion. It's not a Wiki, though. -- EugeneEricKim

OmidyarDotNet has a directive called on-include where any workspace page (that's what they call wiki pages) from any OmidyarDotNet group (each group has it's own seperate workspace pages) can be included in any other workspace page (in any group) or in a discussion comment. --TedErnst

[TikiWiki CMS/Groupware]'s approach.

Other Ideas

Depending on the level of support you want, implementing a full transclusion (an entire page) on a Wiki can range from easy to difficult. The easier the solution, the more likely it will render correctly in Lynx, but less likely it will render correctly in browsers in general.

One way would be just to do a text dump of the page into the middle of your document. e.g. (in Perlish)

    $renderedOutput .= `lynx -source $TranscludedURL`;

However, this will result in malformed HTML.

Another way would be to use an IFRAME (inline frame), which is definitely not LynxFriendly and is only available on newer browsers. e.g.

    <IFRAME WIDTH="100%" HEIGHT="100%" FRAMEBORDER="0" SRC="$TranscludedURL"></IFRAME>

;;

Another way would be to use full blown frames. This is very complicated, but is more likely the correct thing to do.

The overall strategy is to break the page into rendered chunks, with each chunk being in its own frame. Chunks would be delineated by transclusions. So, for something like

MARK 1: This is some body text.

!TRANSCLUSION http://hyperpolis.com

MARK 2: This is some more body text.

I'm talking a bit more.

!TRANSCLUSION http://usemod.com

MARK 3: some more babbling.

there would be five chunks:

MARK 1 to just before the first transclusion;
The first transclusion;
MARK 2 to just before the second transclusion;
The second transclusion;
MARK 3 to the end.

You will need to create an interface into your script that can render chunks of a page at a time in order to do this.

The initial page load, say wiki?TranscludedPage? would output a frameset. In Perl, it might look like

   # Output a set of "*,*,*,*" etc. one * per row
   $rows = ",*" x $NumberOfChunks;
   $rows =~ s/^.//;                # Remove first comma

   $renderedOutput .= << EOF;
<HTML>
<FRAMESET ROWS="$rows">
EOF

   for( $i = 0; $i < $NumberOfChunks; $i++ )
   {
       $renderedOutput .= "    <FRAME SRC=\"wiki?TranscludedPage&chunk=$i\">\n";
   }

   $renderedOutput .= "</FRAMESET>\n</HTML>"

And then, when you encounterd a chunk parameter, you would only output the text of that chunk.

You will also have to emit TARGET parameters in your internal links to break out of the frameset:

    <A HREF="wiki?AnotherPage" TARGET="_top">AnotherPage</A>

Contributors: SunirShah

Well, that's the first revision. I don't think I've covered all the issues though. --ss

[CategoryWikiTechnology]

Discussion

I hate to keep being the negative one, but allowing arbitrary transclusion (either server or browser-based) is nearly equivalent to allowing arbitrary HTML. (One simply has to have a remote web page (which could be a full-HTML enabled wiki) with the bad HTML.) See RawHtmlWiki for possible problems.

Server-based transclusion could filter out some attacks (by removing "dangerous" tags like applets). Server-based transclusion opens other problems, such as copyright, liability for illegal material, and possibly masking of DenialOfService attacks. (Even browser-based transclusion could have legal problems--it is an automatically-executed instruction unlike a link that requires human intervention to select.)

I'd like to believe in transclusion, but I still don't see compelling benefits. Server-based (full-text) transclusion of remote sites is unlikely to be part of my future work. I'm still interested in the possibilities of server-based local transclusion (mostly for ViewPoint), and maybe browser-based frame-transclusion for remote sites (possibly from a list of trusted sites, like other wikis). --CliffordAdams

I don't think it's a big problem. If the foreign site is damaging, people will remove the link. CommunitySolution and all that. Transcluding on a wiki also gives you the ability to search for backlinks as PeterMerel points out on the MuWebWeb link above. -- SunirShah

I think transclusion, albeit internal, could be a great thing to implement. One benefit is to ease the making of a printout of a wiki. Through InternalTransclusion one could let the separate pages live and be edited, and then one could make a printout by having a single page transcluding the sections wanted. If this is combined with editing of headers I think it would be a very interesting concept. --WorldWideWiki:EvenHolen

[I may move most of the following to a page like HtmlSecurity?.

That would be a good idea, but the name would be misleading (or not clear enough). I humbly suggest "SecuringForeignHtml?". ]

By not enabling raw-HTML features, I am removing all dangerous pages before they are created. How's that for a CommunitySolution? More seriously, how would the following disclaimer sound:

: Warning! By visiting any page of this [hypothetical transclusion wiki], you are risking exposure to any HTML-exploitable security flaws. The consequences of these flaws may range from exposure of private information to destruction of all data on your browsing system. To avoid these flaws, either use a fully secure browser, or at least turn off features such as Java and Javascript.

...If that sounds OK to you, then I'll help you install your own separate wiki, and possibly even help you code a transclusion feature. I won't visit it once it's public, however. I don't like automatically executing arbitrary content from arbitrary people. (I don't visit MetaBaby any more for this reason.)

It is important to me that browsing a wiki should be as "safe" as browsing other public discussion sites. In a better world, arbitrary HTML would not be "dangerous"--it's worst effect would be to display unpleasant text or images. If we find such a world, I'll reconsider raw-HTML features.

Even without security problems, I don't really care to support a full-HTML community. I want to emphasize well-written text (with an occasional picture) rather than the all-too-common "pretty HTML" site. This is a personal preference, and other people may disagree. (That's why there's more than one site on the Internet. :-) --CliffordAdams

You bring up an excellent point. That is precisely why we have our own WikiSyntax (which differs from every other wiki!). -- SunirShah

The other wikis are all wrong. Things will be so much simpler when they follow the One True Way (mine). --CliffordAdams ("What's that about snowballs and Hell?")

Well, WikiPedia's WikiSyntax is based on UseModWiki... and more and more people are getting into wikis through Wikipedia... -- StephenGilbert

Transclusion and backlinks:

Only server-based transclusion would seem to help with backlinks. (Client-based transclusion is not much more than a URL to the wiki.) While it is interesting, I'm concerned about the practicality of loading all transcluded pages in order to perform a search. (Caching the pages seems to defeat much of the purpose of transclusion (to track remote changes), and it could easily require huge amounts of space.) --CliffordAdams

I don't understand this. What problem with backlinks are you solving? Why do you need to perform a search against the foreign content? -- SunirShah

This section was a reply to your statement: "Transcluding on a wiki also gives you the ability to search for backlinks as PeterMerel points out on the MuWebWeb link above." I was pointing out that transclusion doesn't seem to affect backlinks (unless one uses server-based transclusion, which has caching problems). Perhaps I misunderstood. --CliffordAdams

: What's the status on transclusion? Should I try it, and with what syntax? -- JerryMuelver

It's easy enough to implement naively. It's even easier on ZwikiClone. See http://www.zwiki.org and note the transclusion of SimonsLog? onto the front page.

Examples Wanted:

I'm curious about whether there are any good examples of transclusion on the net. The best uses I've seen so far are essentially cheap portal sites which merge other sites (rather than providing their own content). Does anyone have better examples, or even interesting stories of what they would *like* to see?

A very interesting example is a service on the net that takes pages from certain sites (like InternetMovieDatabase?), reads a page onna movie and then transform that HTML page to XML. So you request a special URL on that service and get the IMDB data in a form that makes it easy to store to a database, for example. Sort of a TransformingTransclusion?. -- JürgenHermann

WikiPedia, and by extension most MediaWiki wikis, are now very good examples of limited internal transclusion. Mostly template-based, but the transclusion syntax allows a person to transclude just about any text anywhere (it has different systems for other media, if I recall correctly). -- LeeDavisThalbourne

A while back, I started something like a "navigation engine" at http://browse.name/ (involving frames). I haven't done mery much with it in the past couple months, but can see that many of the issues addressed here might also apply to the navigation of that site. Mostly: are users *confused* about which content they are viewing? -- nmw

MediaWiki has now a much more developed transclusion: you can enable [ScaryTranscluding] to transclude any page from a wiki in the InterWiki table. Combined with [Labeled Section Transclusion], this let us transclude chosen parts of any page of any other wiki (and of the wiki itself). An example is the possibility to read side by side two language version of a text on WikiSource?, see WikiPedia:s:Main_Page: try and click ⇔ in the "In other languages" section of the page.

TransclusionLoop

If you transclude WikiOne into WikiTwo an then WikiTwo into WikiOne, when loading WikiOne, WikiTwo would be loaded and would try to load WikiOne again, creating a TransclusionLoop.

When implementing TransClusion there should be some kind of loaded wikis list, to prevent them from loading again.

This would also prevent a wiki from transcluding itself.

The solution to the recursion problem is the same as any graph traversal where you might have a cycle. Just maintain a stack of visited nodes. When encountering a new node, scan the stack for that new node. If it exists in the stack, then it's already been visited in this arc, so don't process that new node.

Wiki transclusion scenario:

Wiki #1 - home wiki for the Foo Wiki engine, which is just another Wiki implementation by Jack Foo

Wiki #2 - wiki for friends of Jack Foo, focuses mostly on social implications of Internet techologies, architectural issues with building Wikis, and rice markets in China

Wiki #3 - user documentation wiki for the Magical Wiki engine, written by John Magic, which extends the Foo Wiki engine, but which has advanced documentation authoring features, such as sophisticated table-of-contents tags, page trail mechanisms, advanced user preferences for HTML style sheets

Wiki #4 - wiki for Jill Public, who is writing a paper called "A Case Study: Using the Magical Wiki to Draft Tariff Policies for Asian Rice Importers"

Some one using Wiki #4 might want to transclude pages from Wiki #2. When Wiki #4 has a link to Wiki #2, it might do a server-side transclusion. When the user hits the link, they stay in Wiki #4, but Wiki #4 uses the WikiWikiTransportProtocol to get the raw data page and linkable tags from Wiki #2. Wiki #4 also scans the raw data page for links back to pages that reside on Wiki #4. Wiki #4 does the raw-data-to-HTML formatting logic, which in this case would be easy, since it's just a subset of its own WikiLanguage?.

-- SteveHowell

Wiki #5 - InternalTransclusion: A given wiki about a subject contains several pages, and then one wants to make a printout. By using transclusion one could easily collapse/gather all the pages into a big one, which then could be printed. And to make a proper printout one might transfrom internal link (between transcluded pages) into visible references, i.e. See 1.6, and all others references visible, i.e. [123], and then included in a long list at the end. It's also possible to shift header levels in the transcluded files, so one could maintain a proper heading level system. (That is h1 and h2 in a transcluded file could be shifted to be h3 and h4 (if that would be appropriate according to context)

There are lots of different Wikis.

A lot of these wikis have duplicated information. The prime example is that a lot of wikis take entries from WikiPedia and copy the information to their own wiki.

It would be useful to be able to use php includes to reference entries on other wikis. This would allow for a more dynamic and active form of copying information. If a WikiPedia entry about soap was relevant to meatball, I would write <?php include("http://en.wikipedia.org/wiki/Soap");> on an entry here at meatball, and this page would appear the same as the WikiPedia entry. If the WikiPedia entry changed, so would this one. If I want to edit the information, I go to the source, to WikiPedia, and edit it there.

I believe this is a powerful concept that will make it much easier for a small number of people to run a very good wiki, in terms of content at least, if not in terms of community. --ChrisAnderson?

Not all wiki engines use PHP. The generalized form of what you're talking about is TransClusion, and many wikis already implement it (or have patches to implement it). Some wikis only support InternalTransclusion, while others allow you to transclude arbitrary web pages. The latter is more difficult for various reasons.

You could also consider NearLink or a simple redirect to the page in question, unless you want to stick your site's headers and footers on the other page, which is frowned upon.

How about this example: Many of us have UserPages? on multiple wikis. I do. All of these UserPages? are more or less the same. Wouldn't it be nice if I could add something to one of my UserPages?, and this updates all of them? --ChrisAnderson?

: I've thought that with some of the wikis that now support RSS, to use RSS for my user page so that all could be read off of one. -- ChristopherAllen

: This could be better solved by letting people to link their wiki user name to a choosen page outside of the wiki. -- (like my signature here)[ZbigniewLukasiak]

Both NearLinks and SisterSites address this. A possible limitation is that those require a certain set of wikis to be commonly recognized as "reference" wikis. Alternatives would be to allow PageRedirect to send you to external sites, or to allow PageAliases that reference external sites.

: There is a subtle difference - both are about joining NameSpaces, so they are a bit heavyweight and require some political action - the recognition as "reference" wikis and some administrative action - changing the config files. What I propose is a bit different, I'd like to have the ability to inform the wiki system that my normal CamelCase signature ZbigniewLukasiak should be rendered as the external link: [ZbigniewLukasiak]. This is just a case by case thing, a shortcut, not a wholesale joining of namespaces.

It can be problematic reversing such things. For example, if I change UseRealNames to redirect to a random porn site, it becomes non-trivial to edit that page and revert it, unless a lot of care is put in to the engine. On this site, undoing a redirect is hard as the VersionHistory becomes inaccessible.

How does your proposal differ from a PageRedirect referencing an external site, by the way? -- ChrisPurcell

Remember that not all sites offer the RightToInclude in their provisions for reuse. --MartinHarper

A problem I see with TransClusion is that it depends on the cooperation of the transcluded. If the site is uncooperative, as it might be if the TransClusion is used for the FairUse practices of criticism and parody, the target site can easily break the transclusion. An uncooperative target could change the meaning of the context around the TransClusion, for instance, taking the example above, if Sunir desired, he could replace the image referred to by the IMG tag with a Microsoft logo. As in:

I love since it reminds me of the color of my wife's eyes. --DavidForrest

TransClusion

Copyright

Implementations

Other Ideas

Related

Discussion

Discussion