Archives/gnosygnu_xowa
1
0
Fork 0
mirror of https://github.com/gnosygnu/xowa.git synced 2024-10-27 20:34:16 +00:00
Code Issues Projects Releases Wiki Activity
e9517cf50d
gnosygnu_xowa/home/wiki/Help/Features/Search/Build.html
gnosygnu e9517cf50d v3.10.4 postponement
2016-10-23 21:41:50 -04:00

318 lines
14 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html dir="ltr">
<head>
<meta http-equiv="content-type" content="text/html;charset=UTF-8" />
<title>Help/Features/Search/Build - XOWA</title>
<link rel="shortcut icon" href="https://gnosygnu.github.io/xowa/xowa_logo.png" />
<link rel="stylesheet" href="https://gnosygnu.github.io/xowa/xowa_common.css" type="text/css">
</head>
<body class="mediawiki ltr sitedir-ltr ns-0 ns-subject skin-vector action-submit vector-animateLayout" spellcheck="false">
<div id="mw-page-base" class="noprint"></div>
<div id="mw-head-base" class="noprint"></div>
<div id="content" class="mw-body">
<h1 id="firstHeading" class="firstHeading"><span>Help/Features/Search/Build</span></h1>
<div id="bodyContent" class="mw-body-content">
<div id="siteSub">From XOWA: the free, open-source, offline wiki application</div>
<div id="contentSub"></div>
<div id="mw-content-text" lang="en" dir="ltr" class="mw-content-ltr">
<p>
XOWA builds search indexes in a multi-step process
</p>
<p>
<br>
</p>
<div id="toc" class="toc">
<div id="toctitle">
<h2>
Contents
</h2>
</div>
<ul>
<li class="toclevel-1 tocsection-1">
<a href="#Overview"><span class="tocnumber">1</span> <span class="toctext">Overview</span></a>
</li>
<li class="toclevel-1 tocsection-2">
<a href="#Process"><span class="tocnumber">2</span> <span class="toctext">Process</span></a>
</li>
<li class="toclevel-1 tocsection-3">
<a href="#Build_process"><span class="tocnumber">3</span> <span class="toctext">Build process</span></a>
<ul>
<li class="toclevel-2 tocsection-4">
<a href="#Download_the_XOWA_search_databases_from_archive.org"><span class="tocnumber">3.1</span> <span class="toctext">Download the XOWA search databases from archive.org</span></a>
</li>
<li class="toclevel-2 tocsection-5">
<a href="#Use_page-length_instead_of_PageRank"><span class="tocnumber">3.2</span> <span class="toctext">Use page-length instead of PageRank</span></a>
</li>
<li class="toclevel-2 tocsection-6">
<a href="#Use_PageRank_but_limit_to_1_iteration"><span class="tocnumber">3.3</span> <span class="toctext">Use PageRank but limit to 1 iteration</span></a>
</li>
<li class="toclevel-2 tocsection-7">
<a href="#Use_PageRank_but_limit_to_1000_iteration"><span class="tocnumber">3.4</span> <span class="toctext">Use PageRank but limit to 1000 iteration</span></a>
</li>
</ul>
</li>
<li class="toclevel-1 tocsection-8">
<a href="#Other_notes"><span class="tocnumber">4</span> <span class="toctext">Other notes</span></a>
</li>
</ul>
</div>
<h2>
<span class="mw-headline" id="Overview">Overview</span>
</h2>
<p>
As of v3.3.4.1, XOWA has a new search engine. This uses the same search functionality that is in XOWA Android.
</p>
<p>
Unfortunately, this search engine is not backward compatible with old wikis. To use it, you must do one of the following:
</p>
<ul>
<li>
Import a new wiki with v3.3.4.1. The new search-databases are built automatically.
</li>
<li>
Upgrade an existing wiki with v3.3.4.1. You can upgrade with the following process
<ul>
<li>
Go to <a href="http://xowa.org/home/wiki/Dashboard/Wiki_maintenance.html" id="xolnki_2" title="Dashboard/Wiki maintenance">Dashboard/Wiki_maintenance</a>
</li>
<li>
Find your wiki in the list
</li>
<li>
Click the <code>Search</code> button in the same row
</li>
<li>
Click <code>Run Script</code>
</li>
</ul>
</li>
</ul>
<p>
<br>
</p>
<h2>
<span class="mw-headline" id="Process">Process</span>
</h2>
<p>
XOWA will generate a search index during the wiki import process. The following steps occur:
</p>
<ul>
<li>
XOWA reads every page title and breaks it up into words
</li>
<li>
XOWA stores this data in an <a href="https://en.wikipedia.org/wiki/Inverted_index" rel="nofollow" class="external text">inverted index</a>. From a database standpoint, they are placed in two database tables called search_word and search_link.
</li>
<li>
XOWA then downloads a list of pagelinks from Wikimedia's dump servers. For example, for 2016-03 English Wikipeda, the link is <a href="http://dumps.wikimedia.org/enwiki/20160305/enwiki-20160305-pagelinks.sql.gz" rel="nofollow" class="external free">http://dumps.wikimedia.org/enwiki/20160305/enwiki-20160305-pagelinks.sql.gz</a>
</li>
<li>
XOWA then parses this data and calculates <a href="https://en.wikipedia.org/wiki/PageRank" rel="nofollow" class="external text">PageRank</a> based on which page links to which page
</li>
<li>
XOWA then applies a series of calculations to come up with a page score for each page. For more info, see <a href="http://xowa.org/home/wiki/Help/Features/Search/Score.html" id="xolnki_3" title="Help/Features/Search/Score">Help/Features/Search/Score</a>
</li>
</ul>
<p>
<br>
</p>
<h2>
<span class="mw-headline" id="Build_process">Build process</span>
</h2>
<p>
Due to the nature of the PageRank algorithm, a lot of additional time and disk-space is needed. These requirements are especially dramatic for English Wikipedia:
</p>
<ul>
<li>
<b>125+ GB hard disk space needed</b>: The pagelinks dump is compressed at 4.7 GB (.gz), expands to 40 GB (.sql) and will require a scratch space of 80 GB (.sqlite3).
</li>
<li>
<b>8+ hours of processing time needed</b>: The PageRank algorithm is computationally expensive on three fronts:
<ul>
<li>
English Wikipedia has 16.3 million pages
</li>
<li>
Each page links to each other through over 1 billion links
</li>
<li>
PageRank needs approximately 20 iterations to completely rank all pages.
</li>
</ul>
</li>
</ul>
<dl>
<dd>
<dl>
<dd>
Note, on a <a href="http://xowa.org/home/wiki/Help/Admin/Environment/Machine.html" id="xolnki_4" title="Help/Admin/Environment/Machine">machine with a fast processor and an SSD</a> this process will <b>only</b> take about 2 hours.
</dd>
</dl>
</dd>
</dl>
<p>
With this in mind, XOWA offers the following options:
</p>
<h3>
<span class="mw-headline" id="Download_the_XOWA_search_databases_from_archive.org">Download the XOWA search databases from archive.org</span>
</h3>
<p>
Monthly versions of English Wikipedia's search databases will be posted to <a href="https://archive.org/edit/Xowa_enwiki_latest" rel="nofollow" class="external free">https://archive.org/edit/Xowa_enwiki_latest</a> . You can just download a 2 GB dump of these databases and replace your copies.
</p>
<h3>
<span class="mw-headline" id="Use_page-length_instead_of_PageRank">Use page-length instead of PageRank</span>
</h3>
<p>
XOWA can use page-length and skip the pagelinks download (125+ GB) as well as the PageRank running time (8+ hours). However the generated results will not be as accurate as PageRank. Specifically, long pages like "List of ...." will have a high page score.
</p>
<p>
To use this option, do the following:
</p>
<ul>
<li>
Go to <a href="http://xowa.org/home/wiki/Options/Import.html" id="xolnki_5" title="Options/Import" class="xowa-visited">Options/Import</a>
</li>
<li>
Change "PageRank iteration count" to 0
</li>
<li>
Save the page
</li>
<li>
Import the wiki.
</li>
</ul>
<p>
Note that 0 is the default value for this option.
</p>
<h3>
<span class="mw-headline" id="Use_PageRank_but_limit_to_1_iteration">Use PageRank but limit to 1 iteration</span>
</h3>
<p>
This option will still require a lot of disk space, but will limit the running time to a few hours. To use this option, do the same as above, but change "PageRank iteration count" to 1.
</p>
<h3>
<span class="mw-headline" id="Use_PageRank_but_limit_to_1000_iteration">Use PageRank but limit to 1000 iteration</span>
</h3>
<p>
This option will create the full version of PageRank search indexes. To use this option, do the same as above, but change "PageRank iteration count" to 1000.
</p>
<p>
<br>
</p>
<h2>
<span class="mw-headline" id="Other_notes">Other notes</span>
</h2>
<ul>
<li>
<b>Test with Simple Wikipedia</b>: The full PageRank process only takes about 20 minutes and requires no more than another 1 GB.
</li>
<li>
<b>Recreate through <a href="http://xowa.org/home/wiki/Dashboard/Wiki_maintenance.html" id="xolnki_6" title="Dashboard/Wiki maintenance">Dashboard/Wiki_maintenance</a></b>: A search index is built when a wiki is first created. You can also recreate it at <a href="http://xowa.org/home/wiki/Dashboard/Wiki_maintenance.html" id="xolnki_7" title="Dashboard/Wiki maintenance">Dashboard/Wiki_maintenance</a>
</li>
</ul>
</div>
</div>
</div>
<div id="mw-head" class="noprint">
<div id="left-navigation">
<div id="p-namespaces" class="vectorTabs">
<h3>Namespaces</h3>
<ul>
<li id="ca-nstab-main" class="selected"><span><a id="ca-nstab-main-href" href="index.html">Page</a></span></li>
</ul>
</div>
</div>
</div>
<div id='mw-panel' class='noprint'>
<div id='p-logo'>
<a style="background-image: url(https://gnosygnu.github.io/xowa/xowa_logo.png);" href="http://xowa.org/" title="Visit the main page"></a>
</div>
<div class="portal" id='xowa-portal-home'>
<h3>XOWA</h3>
<div class="body">
<ul>
<li><a href="http://xowa.org/index.html" title='Visit the main page'>Main page</a></li>
<li><a href="http://xowa.org/screenshots.html" title='See screenshots of XOWA'>Screenshots</a></li>
<li><a href="https://www.youtube.com/watch?v=q0qbXYXEH6M" title="See a video of XOWA Desktop in action">Video</a></li>
<li><a href="http://xowa.org/home/wiki/Help/Download_XOWA.html" title='Download the XOWA application'>Download XOWA</a></li>
<li><a href="http://xowa.org/home/wiki/Dashboard/Image_databases.html" title='Download offline wikis and image databases'>Download wikis</a></li>
</ul>
</div>
</div>
<div class="portal" id='xowa-portal-started'>
<h3>Getting started</h3>
<div class="body">
<ul>
<li><a href="http://xowa.org/home/wiki/App/Setup/System_requirements.html" title='Get XOWA&apos;s system requirements'>Requirements</a></li>
<li><a href="http://xowa.org/home/wiki/App/Setup/Installation.html" title='Get instructions for installing XOWA'>Installation</a></li>
<li><a href="http://xowa.org/home/wiki/App/Import/Simple_Wikipedia.html" title='Learn how to set up Simple Wikipedia'>Simple Wikipedia</a></li>
<li><a href="http://xowa.org/home/wiki/App/Import/English_Wikipedia.html" title='Learn how to set up English Wikipedia'>English Wikipedia</a></li>
<li><a href="http://xowa.org/home/wiki/App/Import/Other_wikis.html" title='Learn how to set up other Wikipedias'>Other Wikipedias</a></li>
</ul>
</div>
</div>
<div class="portal" id='xowa-portal-android'>
<h3>Android</h3>
<div class="body">
<ul>
<li><a href="http://xowa.org/home/wiki/Android/Setup.html" title='Setup XOWA on your Android device'>Setup</a></li>
<li><a href="https://www.youtube.com/watch?v=jsMTBxGweUw" title="See a video of XOWA Android in action">Video</a></li>
</ul>
</div>
</div>
<div class="portal" id='xowa-portal-help'>
<h3>Help</h3>
<div class="body">
<ul>
<li><a href="http://xowa.org/home/wiki/Help/About.html" title='Get more information about XOWA'>About</a></li>
<li><a href="http://xowa.org/home/wiki/Help/Contents.html" title='View a list of help topics'>Contents</a></li>
<li><a href="http://xowa.org/home/wiki/Help/Media.html" title='Read what others have written about XOWA'>Media</a></li>
<li><a href="http://xowa.org/home/wiki/Help/Feedback.html" title='Questions? Comments? Leave feedback for XOWA'>Feedback</a></li>
</ul>
</div>
</div>
<div class="portal" id='xowa-portal-blog'>
<h3>Blog</h3>
<div class="body">
<ul>
<li><a href="http://xowa.org/home/wiki/Blog.html" title='Follow XOWA''s development process'>Current</a></li>
</ul>
</div>
</div>
<div class="portal" id='xowa-portal-links'>
<h3>Links</h3>
<div class="body">
<ul>
<li><a href="http://dumps.wikimedia.org/backup-index.html" title="Get wiki datababase dumps directly from Wikimedia">Wikimedia dumps</a></li>
<li><a href="https://archive.org/search.php?query=xowa" title="Search archive.org for XOWA files">XOWA @ archive.org</a></li>
<li><a href="http://en.wikipedia.org" title="Visit Wikipedia (and compare to XOWA!)">English Wikipedia</a></li>
</ul>
</div>
</div>
<div class="portal" id='xowa-portal-donate'>
<h3>Donate</h3>
<div class="body">
<ul>
<li><a href="https://archive.org/donate/index.php" title="Support archive.org!">archive.org</a></li><!-- listed first due to recent fire damages: http://blog.archive.org/2013/11/06/scanning-center-fire-please-help-rebuild/ -->
<li><a href="https://donate.wikimedia.org/wiki/Special:FundraiserRedirector" title="Support Wikipedia!">Wikipedia</a></li>
<!-- <li><a href="" title="Support XOWA! (but only after you've supported archive.org and Wikipedia)">XOWA</a></li> -->
</ul>
</div>
</div>
</div>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink