mirror of
https://github.com/gnosygnu/xowa.git
synced 2025-05-30 22:14:38 +00:00
376 lines
16 KiB
HTML
376 lines
16 KiB
HTML
<!DOCTYPE html>
|
|
<html dir="ltr">
|
|
<head>
|
|
<meta http-equiv="content-type" content="text/html;charset=UTF-8" />
|
|
<title>App/Xtn/Import/Dansguardian - XOWA</title>
|
|
<link rel="shortcut icon" href="https://gnosygnu.github.io/xowa/xowa_logo.png" />
|
|
<link rel="stylesheet" href="https://gnosygnu.github.io/xowa/xowa_common.css" type="text/css">
|
|
|
|
</head>
|
|
<body class="mediawiki ltr sitedir-ltr ns-0 ns-subject skin-vector action-submit vector-animateLayout" spellcheck="false">
|
|
<div id="mw-page-base" class="noprint"></div>
|
|
<div id="mw-head-base" class="noprint"></div>
|
|
<div id="content" class="mw-body">
|
|
<h1 id="firstHeading" class="firstHeading"><span>App/Xtn/Import/Dansguardian</span></h1>
|
|
<div id="bodyContent" class="mw-body-content">
|
|
<div id="siteSub">From XOWA: the free, open-source, offline wiki application</div>
|
|
<div id="contentSub"></div>
|
|
<div id="mw-content-text" lang="en" dir="ltr" class="mw-content-ltr">
|
|
|
|
<p>
|
|
XOWA allows custom creation of wikis by either excluding words or including words. The system is based on the <a href="http://dansguardian.org/" rel="nofollow" class="external text">Dansguardian</a> format: an open-source (GPLv2) system for filtering web-pages.
|
|
</p>
|
|
<div id="toc" class="toc">
|
|
<div id="toctitle">
|
|
<h2>
|
|
Contents
|
|
</h2>
|
|
</div>
|
|
<ul>
|
|
<li class="toclevel-1 tocsection-1">
|
|
<a href="#Options"><span class="tocnumber">1</span> <span class="toctext">Options</span></a>
|
|
</li>
|
|
<li class="toclevel-1 tocsection-2">
|
|
<a href="#Phraselists"><span class="tocnumber">2</span> <span class="toctext">Phraselists</span></a>
|
|
<ul>
|
|
<li class="toclevel-2 tocsection-3">
|
|
<a href="#Location"><span class="tocnumber">2.1</span> <span class="toctext">Location</span></a>
|
|
</li>
|
|
<li class="toclevel-2 tocsection-4">
|
|
<a href="#Format"><span class="tocnumber">2.2</span> <span class="toctext">Format</span></a>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li class="toclevel-1 tocsection-5">
|
|
<a href="#Import_process"><span class="tocnumber">3</span> <span class="toctext">Import process</span></a>
|
|
</li>
|
|
<li class="toclevel-1 tocsection-6">
|
|
<a href="#Scoring"><span class="tocnumber">4</span> <span class="toctext">Scoring</span></a>
|
|
<ul>
|
|
<li class="toclevel-2 tocsection-7">
|
|
<a href="#Basic"><span class="tocnumber">4.1</span> <span class="toctext">Basic</span></a>
|
|
</li>
|
|
<li class="toclevel-2 tocsection-8">
|
|
<a href="#Multiplicity"><span class="tocnumber">4.2</span> <span class="toctext">Multiplicity</span></a>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li class="toclevel-1 tocsection-9">
|
|
<a href="#Exclusion"><span class="tocnumber">5</span> <span class="toctext">Exclusion</span></a>
|
|
</li>
|
|
<li class="toclevel-1 tocsection-10">
|
|
<a href="#Inclusion"><span class="tocnumber">6</span> <span class="toctext">Inclusion</span></a>
|
|
</li>
|
|
<li class="toclevel-1 tocsection-11">
|
|
<a href="#Manual_inclusion_.2F_exclusion"><span class="tocnumber">7</span> <span class="toctext">Manual inclusion / exclusion</span></a>
|
|
</li>
|
|
<li class="toclevel-1 tocsection-12">
|
|
<a href="#Other_notes"><span class="tocnumber">8</span> <span class="toctext">Other notes</span></a>
|
|
<ul>
|
|
<li class="toclevel-2 tocsection-13">
|
|
<a href="#Performance"><span class="tocnumber">8.1</span> <span class="toctext">Performance</span></a>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
</div>
|
|
<h2>
|
|
<span class="mw-headline" id="Options">Options</span>
|
|
</h2>
|
|
<p>
|
|
See <a href="http://xowa.org/home/wiki/Options/Import_Dansguardian.html" id="xolnki_2" title="Options/Import Dansguardian">Options/Import_Dansguardian</a>
|
|
</p>
|
|
<h2>
|
|
<span class="mw-headline" id="Phraselists">Phraselists</span>
|
|
</h2>
|
|
<h3>
|
|
<span class="mw-headline" id="Location">Location</span>
|
|
</h3>
|
|
<ul>
|
|
<li>
|
|
Phraselist files are located at <code>/xowa/bin/any/xowa/cfg/bldr/filter/wiki_name/dansguardian</code>. For example, on a Windows system, a phraselist file for simple.wikipedia.org can be placed at <code>C:\xowa\bin\any\xowa\cfg\bldr\filter\simple.wikipedia.org\dansguardian\phraselist1.txt</code>
|
|
</li>
|
|
<li>
|
|
Phraselist files can be placed in sub-directories for grouping purposes. For example, files can be placed at C:\xowa\bin\any\xowa\cfg\bldr\filter\simple.wikipedia.org\dansguardian\group1\phraselist1.txt and C:\xowa\bin\any\xowa\cfg\bldr\filter\simple.wikipedia.org\dansguardian\group2\phraselist2.txt
|
|
</li>
|
|
</ul>
|
|
<h3>
|
|
<span class="mw-headline" id="Format">Format</span>
|
|
</h3>
|
|
<p>
|
|
Phraselist files are plain text files with the following format:
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
Each line is either a rule or a comment
|
|
</li>
|
|
<li>
|
|
Comments start with the hash sign: # . For example <code># this is a comment</code>
|
|
</li>
|
|
<li>
|
|
Each rule has one or more words enclosed in angle brackets (< >) and separated by commas. For example, <code>< earth >,< mars ></code>
|
|
</li>
|
|
<li>
|
|
Each rule ends with a score also enclosed in angle brackets. For example, <code><70></code>
|
|
</li>
|
|
</ul>
|
|
<h2>
|
|
<span class="mw-headline" id="Import_process">Import process</span>
|
|
</h2>
|
|
<p>
|
|
Phraselists are applied during import. The following process occurs:
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
The import starts for a wiki
|
|
</li>
|
|
<li>
|
|
All phraselists for the wiki are loaded into memory.
|
|
</li>
|
|
<li>
|
|
Each article's wikitext is analyzed by the phraselists and generates a score.
|
|
<ul>
|
|
<li>
|
|
The article's "title" is not analyzed. Article titles generally have only one or two words, and are not useful for phraselist matching
|
|
</li>
|
|
<li>
|
|
The html is not analyzed. Note that this would slow down the import process dramatically. For example, for English Wikipedia, wikitext would only slow down the process from 2 hours 40 minutes to 3 hours. HTML would slow it down to 70 hours.
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<p>
|
|
The entire wikitext is scanned including:
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
Html tags: <img some attributes/>
|
|
</li>
|
|
<li>
|
|
Urls: https://upload.wikimedia.org
|
|
</li>
|
|
<li>
|
|
Comments
|
|
</li>
|
|
</ul>
|
|
<h2>
|
|
<span class="mw-headline" id="Scoring">Scoring</span>
|
|
</h2>
|
|
<h3>
|
|
<span class="mw-headline" id="Basic">Basic</span>
|
|
</h3>
|
|
<p>
|
|
A rule is matched if any part of the wikitext contains the words in the ruletext.
|
|
</p>
|
|
<p>
|
|
For example, let's says we wanted to build up a phraselist that allowed us to build a wiki without any astronomy articles. We could use something like the following:
|
|
</p>
|
|
<pre>
|
|
< planet ><50>
|
|
< earth >,< planet ><30>
|
|
</pre>
|
|
<p>
|
|
Now consider these short sample articles:
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
An article with just the word "planet" would have a score of 50
|
|
</li>
|
|
<li>
|
|
An article with the words "earth planet something" would have a score of 80; 50 for matching "planet" and 30 for matching "earth" and " planet "
|
|
</li>
|
|
<li>
|
|
An article with just the word "earth" would have a score of 0. It needs to have the word "planet" to get a score of 30
|
|
</li>
|
|
</ul>
|
|
<h3>
|
|
<span class="mw-headline" id="Multiplicity">Multiplicity</span>
|
|
</h3>
|
|
<p>
|
|
Rules scores are multiplied based on occurrences.
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
For example, if an article has the text "planet planet planet" then its score would be 150, not 50, because it matches the "planet" rule 3 times
|
|
</li>
|
|
<li>
|
|
Similarly "earth planet something earth planet something earth planet something" would have a score of 240 because it matches "earth planet" 3 times (90) and "planet" 3 times (150)
|
|
</li>
|
|
<li>
|
|
However "earth planet something earth planet something earth" only has a score of 160 because it only matches the "earth planet" rule 2 times "60" and the planet rule 2 times (100)
|
|
</li>
|
|
</ul>
|
|
<h2>
|
|
<span class="mw-headline" id="Exclusion">Exclusion</span>
|
|
</h2>
|
|
<p>
|
|
By default, anything that matches a rule (has a score > 0) will be excluded. Note that this exclude number can be raised from 0 to something higher like 100. See <a href="http://xowa.org/home/wiki/Options/Import_Dansguardian.html" id="xolnki_3" title="Options/Import Dansguardian">Options/Import_Dansguardian</a>
|
|
</p>
|
|
<h2>
|
|
<span class="mw-headline" id="Inclusion">Inclusion</span>
|
|
</h2>
|
|
<p>
|
|
The import filter can also be used to build content-specific wikis. For example, let's say you wanted to build a wiki that only <b>includes</b> articles with the words "planet" and "earth planet". The following can be done:
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
Use the same phraselists as above, but negate the numbers:
|
|
</li>
|
|
</ul>
|
|
<pre>
|
|
< planet ><-50>
|
|
< earth >,< planet ><-30>
|
|
</pre>
|
|
<ul>
|
|
<li>
|
|
Change the initial score from 0 to 50
|
|
</li>
|
|
<li>
|
|
Leave the exclude score at 0
|
|
</li>
|
|
</ul>
|
|
<p>
|
|
When running the import, the following will happen:
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
An article that has the words "earth planet something" will have a score of -30: the initial score of 50 plus the rule score of -80. Because -30 is less than the exclude score of 0, it will not be excluded.
|
|
</li>
|
|
<li>
|
|
An article that has the words "a b c" will still have its initial score of 50. Because 50 is greater than the exclude score of 0, it will be excluded
|
|
</li>
|
|
</ul>
|
|
<h2>
|
|
<span class="mw-headline" id="Manual_inclusion_.2F_exclusion">Manual inclusion / exclusion</span>
|
|
</h2>
|
|
<p>
|
|
The filter process also provides a way to list articles that will always be included / excluded, regardless of rule score.
|
|
</p>
|
|
<p>
|
|
For example, to always include the articles Earth and Sun, create the following text file:
|
|
</p>
|
|
<pre>
|
|
Earth
|
|
Sun
|
|
</pre>
|
|
<p>
|
|
Save it to <code>/xowa/bin/any/xowa/cfg/bldr/filter/wiki_name/xowa.title.exclude.txt</code> XOWA will read this list and always exclude the article
|
|
</p>
|
|
<p>
|
|
Similarly, to manually include an article, save the file to /xowa/bin/any/xowa/cfg/bldr/filter/wiki_name/xowa.title.include.txt
|
|
</p>
|
|
<h2>
|
|
<span class="mw-headline" id="Other_notes">Other notes</span>
|
|
</h2>
|
|
<h3>
|
|
<span class="mw-headline" id="Performance">Performance</span>
|
|
</h3>
|
|
<ul>
|
|
<li>
|
|
Note that a phraselist file can have many rules. The number of rules does not significantly slow down the runtime of the import-filter. For example, let's say Simple Wikipedia imports in 3 minutes with 100 rules. If there are 10,000 rules, the import should still take 3 minutes
|
|
</li>
|
|
<li>
|
|
However, the number of rules will affect the amount of memory required by the computer. For example, 100 rules may take 1 MB. 10,000 rules may take 10 MB.
|
|
</li>
|
|
</ul>
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
<div id="mw-head" class="noprint">
|
|
<div id="left-navigation">
|
|
<div id="p-namespaces" class="vectorTabs">
|
|
<h3>Namespaces</h3>
|
|
<ul>
|
|
<li id="ca-nstab-main" class="selected"><span><a id="ca-nstab-main-href" href="index.html">Page</a></span></li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<div id='mw-panel' class='noprint'>
|
|
<div id='p-logo'>
|
|
<a style="background-image: url(https://gnosygnu.github.io/xowa/xowa_logo.png);" href="http://xowa.org/" title="Visit the main page"></a>
|
|
</div>
|
|
<div class="portal" id='xowa-portal-home'>
|
|
<h3>XOWA</h3>
|
|
<div class="body">
|
|
<ul>
|
|
<li><a href="http://xowa.org/index.html" title='Visit the main page'>Main page</a></li>
|
|
<li><a href="http://xowa.org/screenshots.html" title='See screenshots of XOWA'>Screenshots</a></li>
|
|
<li><a href="http://xowa.org/home/wiki/Help/Download_XOWA.html" title='Download the XOWA application'>Download XOWA</a></li>
|
|
<li><a href="http://xowa.org/home/wiki/Dashboard/Image_databases.html" title='Download offline wikis and image databases'>Download wikis</a></li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="portal" id='xowa-portal-started'>
|
|
<h3>Getting started</h3>
|
|
<div class="body">
|
|
<ul>
|
|
<li><a href="http://xowa.org/home/wiki/App/Setup/System_requirements.html" title='Get XOWA's system requirements'>Requirements</a></li>
|
|
<li><a href="http://xowa.org/home/wiki/App/Setup/Installation.html" title='Get instructions for installing XOWA'>Installation</a></li>
|
|
<li><a href="http://xowa.org/home/wiki/App/Import/Simple_Wikipedia.html" title='Learn how to set up Simple Wikipedia'>Simple Wikipedia</a></li>
|
|
<li><a href="http://xowa.org/home/wiki/App/Import/English_Wikipedia.html" title='Learn how to set up English Wikipedia'>English Wikipedia</a></li>
|
|
<li><a href="http://xowa.org/home/wiki/App/Import/Other_wikis.html" title='Learn how to set up other Wikipedias'>Other Wikipedias</a></li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="portal" id='xowa-portal-android'>
|
|
<h3>Android</h3>
|
|
<div class="body">
|
|
<ul>
|
|
<li><a href="http://xowa.org/home/wiki/Android/Setup.html" title='Setup XOWA on your Android device'>Setup</a></li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="portal" id='xowa-portal-help'>
|
|
<h3>Help</h3>
|
|
<div class="body">
|
|
<ul>
|
|
<li><a href="http://xowa.org/home/wiki/Help/About.html" title='Get more information about XOWA'>About</a></li>
|
|
<li><a href="http://xowa.org/home/wiki/Help/Contents.html" title='View a list of help topics'>Contents</a></li>
|
|
<li><a href="http://xowa.org/home/wiki/Help/Media.html" title='Read what others have written about XOWA'>Media</a></li>
|
|
<li><a href="http://xowa.org/home/wiki/Help/Feedback.html" title='Questions? Comments? Leave feedback for XOWA'>Feedback</a></li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="portal" id='xowa-portal-blog'>
|
|
<h3>Blog</h3>
|
|
<div class="body">
|
|
<ul>
|
|
<li><a href="http://xowa.org/home/wiki/Blog.html" title='Follow XOWA''s development process'>Current</a></li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="portal" id='xowa-portal-links'>
|
|
<h3>Links</h3>
|
|
<div class="body">
|
|
<ul>
|
|
<li><a href="http://dumps.wikimedia.org/backup-index.html" title="Get wiki datababase dumps directly from Wikimedia">Wikimedia dumps</a></li>
|
|
<li><a href="https://archive.org/search.php?query=xowa" title="Search archive.org for XOWA files">XOWA @ archive.org</a></li>
|
|
<li><a href="http://en.wikipedia.org" title="Visit Wikipedia (and compare to XOWA!)">English Wikipedia</a></li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="portal" id='xowa-portal-donate'>
|
|
<h3>Donate</h3>
|
|
<div class="body">
|
|
<ul>
|
|
<li><a href="https://archive.org/donate/index.php" title="Support archive.org!">archive.org</a></li><!-- listed first due to recent fire damages: http://blog.archive.org/2013/11/06/scanning-center-fire-please-help-rebuild/ -->
|
|
<li><a href="https://donate.wikimedia.org/wiki/Special:FundraiserRedirector" title="Support Wikipedia!">Wikipedia</a></li>
|
|
<!-- <li><a href="" title="Support XOWA! (but only after you've supported archive.org and Wikipedia)">XOWA</a></li> -->
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
|
|
</div>
|
|
</body>
|
|
</html> |