Archives/gnosygnu_xowa
1
0
Fork 0
mirror of https://github.com/gnosygnu/xowa.git synced 2025-05-31 22:44:34 +00:00
Code Issues Projects Releases Wiki Activity

update Command-Line/dumps

Browse Source
This commit is contained in:
gnosygnu 2017-02-02 11:48:15 -05:00
parent b4a7e4742e
commit f26baacee6
21 changed files with 165 additions and 131 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
home/wiki/App/Import/English_Wikipedia.html
View File

@ -138,7 +138,7 @@
<span class="mw-headline" id="Detailed_start">Detailed start</span> <span class="mw-headline" id="Detailed_start">Detailed start</span>
</h2> </h2>
<p> <p>
See <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_3" title="Wiki setup/English wikis">Wiki_setup/English_wikis</a> See <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_3" title="Wiki setup/English wikis" class="xowa-visited">Wiki_setup/English_wikis</a>
</p> </p>
</div> </div>

7
home/wiki/App/Wiki_types/Wikia.com.html
View File

@ -170,7 +170,12 @@
</h3> </h3>
<ul> <ul>
<li> <li>
Setup Python 2: <a href="https://www.python.org/downloads/" rel="nofollow" class="external free">https://www.python.org/downloads/</a> . For the rest of this walkthrough, we'll assume it's installed at <code>C:\Python2.7</code> Setup Python 2: <a href="https://www.python.org/downloads/" rel="nofollow" class="external free">https://www.python.org/downloads/</a> . For the rest of this walkthrough, we'll assume it's installed at <code>C:\Python2.7</code>
<ul>
<li>
<b>Note that the dumpgenerator.py script will not work with Python 3</b>
</li>
</ul>
</li> </li>
<li> <li>
Download the wikiteam project from <a href="https://github.com/WikiTeam/wikiteam" rel="nofollow" class="external free">https://github.com/WikiTeam/wikiteam</a> For the rest of this walkthrough, we'll assume it's downloaded to <code>C:\WikiTeam</code> Download the wikiteam project from <a href="https://github.com/WikiTeam/wikiteam" rel="nofollow" class="external free">https://github.com/WikiTeam/wikiteam</a> For the rest of this walkthrough, we'll assume it's downloaded to <code>C:\WikiTeam</code>

2
home/wiki/Blog/2016-01.html
View File

@ -416,7 +416,7 @@
The Android app is a major release. It has a Random feature, shows more images, and adds CSS-tweaks. The Android app is a major release. It has a Random feature, shows more images, and adds CSS-tweaks.
</p> </p>
<h3> <h3>
<span class="mw-headline" id="Documentation_for_html-dump_script._See_Dev.2FCommand-line.2FDumps">Documentation for html-dump script. See <a href="http://xowa.org/home/wiki/Dev/Command-line/Dumps.html" id="xolnki_15" title="Dev/Command-line/Dumps">Dev/Command-line/Dumps</a></span> <span class="mw-headline" id="Documentation_for_html-dump_script._See_Dev.2FCommand-line.2FDumps">Documentation for html-dump script. See <a href="http://xowa.org/home/wiki/Dev/Command-line/Dumps.html" id="xolnki_15" title="Dev/Command-line/Dumps" class="xowa-visited">Dev/Command-line/Dumps</a></span>
</h3> </h3>
<p> <p>
This item is self-explanatory. The XOWA Android app is getting more stable, so I felt it would be time to document the generation of the HTML databases. This item is self-explanatory. The XOWA Android app is getting more stable, so I felt it would be time to document the generation of the HTML databases.

2
home/wiki/Blog/2016-06.html
View File

@ -437,7 +437,7 @@
</dl> </dl>
<ul> <ul>
<li> <li>
<b>Requires separate post-processing generation step</b>: The wikitext dumps were automatically generated by downloading an XML dump. The HTML dumps requires another post-processing step that is not simple to run (See: <a href="http://xowa.org/home/wiki/Dev/Command-line/Dumps.html" id="xolnki_20" title="Dev/Command-line/Dumps">Dev/Command-line/Dumps</a>) <b>Requires separate post-processing generation step</b>: The wikitext dumps were automatically generated by downloading an XML dump. The HTML dumps requires another post-processing step that is not simple to run (See: <a href="http://xowa.org/home/wiki/Dev/Command-line/Dumps.html" id="xolnki_20" title="Dev/Command-line/Dumps" class="xowa-visited">Dev/Command-line/Dumps</a>)
</li> </li>
</ul> </ul>
<dl> <dl>

4
home/wiki/Blog/2016-10.html
View File

@ -157,7 +157,7 @@
</p> </p>
<ul> <ul>
<li> <li>
<a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_15" title="Wiki setup/English wikis">Wiki_setup/English_wikis</a> <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_15" title="Wiki setup/English wikis" class="xowa-visited">Wiki_setup/English_wikis</a>
</li> </li>
<li> <li>
<a href="http://xowa.org/home/wiki/Wiki_setup/German_wikis.html" id="xolnki_16" title="Wiki setup/German wikis">Wiki_setup/German_wikis</a> <a href="http://xowa.org/home/wiki/Wiki_setup/German_wikis.html" id="xolnki_16" title="Wiki setup/German wikis">Wiki_setup/German_wikis</a>
@ -238,7 +238,7 @@
</p> </p>
<ul> <ul>
<li> <li>
<a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_18" title="Wiki setup/English wikis">Wiki_setup/English_wikis</a> <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_18" title="Wiki setup/English wikis" class="xowa-visited">Wiki_setup/English_wikis</a>
</li> </li>
<li> <li>
<a href="http://xowa.org/home/wiki/Wiki_setup/German_wikis.html" id="xolnki_19" title="Wiki setup/German wikis">Wiki_setup/German_wikis</a> <a href="http://xowa.org/home/wiki/Wiki_setup/German_wikis.html" id="xolnki_19" title="Wiki setup/German wikis">Wiki_setup/German_wikis</a>

10
home/wiki/Change_log.html
View File

@ -818,7 +818,7 @@
</ul> </ul>
<dl> <dl>
<dd> <dd>
<span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_13" title="Wiki setup/English wikis">Wiki_setup/English_wikis</a> <span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_13" title="Wiki setup/English wikis" class="xowa-visited">Wiki_setup/English_wikis</a>
</dd> </dd>
</dl> </dl>
<h3> <h3>
@ -1122,7 +1122,7 @@
</ul> </ul>
<dl> <dl>
<dd> <dd>
<span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_19" title="Wiki setup/English wikis">Wiki_setup/English_wikis</a> <span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_19" title="Wiki setup/English wikis" class="xowa-visited">Wiki_setup/English_wikis</a>
</dd> </dd>
</dl> </dl>
<ul> <ul>
@ -1426,7 +1426,7 @@
<span style='font-variant:small-caps'>Resolved by</span>: Include latest download central database. <span style='font-variant:small-caps'>Resolved by</span>: Include latest download central database.
</dd> </dd>
<dd> <dd>
<span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_22" title="Wiki setup/English wikis">Wiki_setup/English_wikis</a> <a href="http://xowa.org/home/wiki/Wiki_setup/German_wikis.html" id="xolnki_23" title="Wiki setup/German wikis">Wiki_setup/German_wikis</a> <span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_22" title="Wiki setup/English wikis" class="xowa-visited">Wiki_setup/English_wikis</a> <a href="http://xowa.org/home/wiki/Wiki_setup/German_wikis.html" id="xolnki_23" title="Wiki setup/German wikis">Wiki_setup/German_wikis</a>
</dd> </dd>
</dl> </dl>
<h2> <h2>
@ -1442,7 +1442,7 @@
</ul> </ul>
<dl> <dl>
<dd> <dd>
<span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_24" title="Wiki setup/English wikis">Wiki_setup/English_wikis</a> <span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_24" title="Wiki setup/English wikis" class="xowa-visited">Wiki_setup/English_wikis</a>
</dd> </dd>
</dl> </dl>
<ul> <ul>
@ -2249,7 +2249,7 @@
</ul> </ul>
<dl> <dl>
<dd> <dd>
<span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Dev/Command-line/Dumps.html" id="xolnki_35" title="Dev/Command-line/Dumps">Dev/Command-line/Dumps</a> <span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Dev/Command-line/Dumps.html" id="xolnki_35" title="Dev/Command-line/Dumps" class="xowa-visited">Dev/Command-line/Dumps</a>
</dd> </dd>
</dl> </dl>
<ul> <ul>

2
home/wiki/Change_log/2016.html
View File

@ -2582,7 +2582,7 @@
</p> </p>
<ul> <ul>
<li> <li>
Command-line: Expand instructions for generating HTML dumps. See: <a href="http://xowa.org/home/wiki/Dev/Command-line/Dumps.html" id="xolnki_30" title="Dev/Command-line/Dumps">Dev/Command-line/Dumps</a> Command-line: Expand instructions for generating HTML dumps. See: <a href="http://xowa.org/home/wiki/Dev/Command-line/Dumps.html" id="xolnki_30" title="Dev/Command-line/Dumps" class="xowa-visited">Dev/Command-line/Dumps</a>
</li> </li>
</ul> </ul>
<p> <p>

4
home/wiki/Change_log/v3.10.3.1.html
View File

@ -64,7 +64,7 @@
<span style='font-variant:small-caps'>Resolved by</span>: Include latest download central database. <span style='font-variant:small-caps'>Resolved by</span>: Include latest download central database.
</dd> </dd>
<dd> <dd>
<span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_2" title="Wiki setup/English wikis">Wiki_setup/English_wikis</a> <a href="http://xowa.org/home/wiki/Wiki_setup/German_wikis.html" id="xolnki_3" title="Wiki setup/German wikis">Wiki_setup/German_wikis</a> <span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_2" title="Wiki setup/English wikis" class="xowa-visited">Wiki_setup/English_wikis</a> <a href="http://xowa.org/home/wiki/Wiki_setup/German_wikis.html" id="xolnki_3" title="Wiki setup/German wikis">Wiki_setup/German_wikis</a>
</dd> </dd>
</dl> </dl>
<h2> <h2>
@ -80,7 +80,7 @@
</ul> </ul>
<dl> <dl>
<dd> <dd>
<span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_4" title="Wiki setup/English wikis">Wiki_setup/English_wikis</a> <span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_4" title="Wiki setup/English wikis" class="xowa-visited">Wiki_setup/English_wikis</a>
</dd> </dd>
</dl> </dl>
<ul> <ul>

2
home/wiki/Change_log/v3.10.4.1.html
View File

@ -56,7 +56,7 @@
</ul> </ul>
<dl> <dl>
<dd> <dd>
<span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_2" title="Wiki setup/English wikis">Wiki_setup/English_wikis</a> <span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_2" title="Wiki setup/English wikis" class="xowa-visited">Wiki_setup/English_wikis</a>
</dd> </dd>
</dl> </dl>
<ul> <ul>

2
home/wiki/Change_log/v3.11.2.1.html
View File

@ -70,7 +70,7 @@
</ul> </ul>
<dl> <dl>
<dd> <dd>
<span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_6" title="Wiki setup/English wikis">Wiki_setup/English_wikis</a> <span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Wiki_setup/English_wikis.html" id="xolnki_6" title="Wiki setup/English wikis" class="xowa-visited">Wiki_setup/English_wikis</a>
</dd> </dd>
</dl> </dl>
<h3> <h3>

2
home/wiki/Change_log/v3.9.2.1.html
View File

@ -279,7 +279,7 @@
</ul> </ul>
<dl> <dl>
<dd> <dd>
<span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Dev/Command-line/Dumps.html" id="xolnki_5" title="Dev/Command-line/Dumps">Dev/Command-line/Dumps</a> <span style='font-variant:small-caps'>Links</span>: <a href="http://xowa.org/home/wiki/Dev/Command-line/Dumps.html" id="xolnki_5" title="Dev/Command-line/Dumps" class="xowa-visited">Dev/Command-line/Dumps</a>
</dd> </dd>
</dl> </dl>
<ul> <ul>

109
home/wiki/Dev/Command-line/Dumps.html
View File

@ -68,7 +68,7 @@
<a href="#Requirements"><span class="tocnumber">2</span> <span class="toctext">Requirements</span></a> <a href="#Requirements"><span class="tocnumber">2</span> <span class="toctext">Requirements</span></a>
<ul> <ul>
<li class="toclevel-2 tocsection-3"> <li class="toclevel-2 tocsection-3">
<a href="#commons.wikimedia.org_.28thum"><span class="tocnumber">2.1</span> <span class="toctext">commons.wikimedia.org (thum</span></a> <a href="#commons.wikimedia.org"><span class="tocnumber">2.1</span> <span class="toctext">commons.wikimedia.org</span></a>
</li> </li>
<li class="toclevel-2 tocsection-4"> <li class="toclevel-2 tocsection-4">
<a href="#www.wikidata.org"><span class="tocnumber">2.2</span> <span class="toctext">www.wikidata.org</span></a> <a href="#www.wikidata.org"><span class="tocnumber">2.2</span> <span class="toctext">www.wikidata.org</span></a>
@ -158,7 +158,7 @@
<span class="mw-headline" id="Requirements">Requirements</span> <span class="mw-headline" id="Requirements">Requirements</span>
</h2> </h2>
<h3> <h3>
<span class="mw-headline" id="commons.wikimedia.org_.28thum">commons.wikimedia.org (thum</span> <span class="mw-headline" id="commons.wikimedia.org">commons.wikimedia.org</span>
</h3> </h3>
<p> <p>
You will need the latest version of commons.wikimedia.org. Note that if you have an older version, you will have missing images or wrong size information. You will need the latest version of commons.wikimedia.org. Note that if you have an older version, you will have missing images or wrong size information.
@ -304,6 +304,10 @@
<pre class='code'> <pre class='code'>
app.bldr.pause_at_end_('n'); app.bldr.pause_at_end_('n');
app.scripts.run_file_by_type('xowa_cfg_app'); app.scripts.run_file_by_type('xowa_cfg_app');
app.cfg.set_temp('app', 'xowa.app.web.enabled', 'y');
app.cfg.set_temp('app', 'xowa.bldr.db.layout_size.text', '0');
app.cfg.set_temp('app', 'xowa.bldr.db.layout_size.html', '0');
app.cfg.set_temp('app', 'xowa.bldr.db.layout_size.file', '0');
app.bldr.cmds { app.bldr.cmds {
// build commons database; this only needs to be done once, whenever commons is updated // build commons database; this only needs to be done once, whenever commons is updated
add ('commons.wikimedia.org' , 'util.cleanup') {delete_all = 'y';} add ('commons.wikimedia.org' , 'util.cleanup') {delete_all = 'y';}
@ -391,9 +395,11 @@ app.bldr.cmds {
// cleanup all downloaded files as well as temporary files // cleanup all downloaded files as well as temporary files
add ('simple.wikipedia.org' , 'util.cleanup') {delete_tmp = 'y'; delete_by_match('*.xml|*.sql|*.bz2|*.gz');} add ('simple.wikipedia.org' , 'util.cleanup') {delete_tmp = 'y'; delete_by_match('*.xml|*.sql|*.bz2|*.gz');}
// OBSOLETE: use v2
// v1 html generator // v1 html generator
// parse every page in the listed namespace and gather data on their lnkis. // parse every page in the listed namespace and gather data on their lnkis.
// this step will take the longest amount of time. // this step will take the longest amount of time.
/*
add ('simple.wikipedia.org' , 'file.lnki_temp') { add ('simple.wikipedia.org' , 'file.lnki_temp') {
// save data every # of pages // save data every # of pages
commit_interval = 10000; commit_interval = 10000;
@ -426,13 +432,14 @@ app.bldr.cmds {
hzip_diff = 'y'; hzip_diff = 'y';
} }
} }
*/
// v2 html generator; allows for multi-threaded / multi-machine builds // v2 html generator; allows for multi-threaded / multi-machine builds
/* add ('simple.wikipedia.org' , 'wiki.mass_parse.init') {cfg {ns_ids = '0|4|14|8';}}
add ('simple.wikipedia.org' , 'wiki.mass_parse.init') {cfg {ns_ids = '0|4|14';}}
add ('simple.wikipedia.org' , 'wiki.mass_parse.exec') { add ('simple.wikipedia.org' , 'wiki.mass_parse.exec') {
cfg { cfg {
num_wkrs = 8; load_all_templates = 'y'; cleanup_interval = 50; hzip_enabled = 'y'; hdiff_enabled ='y'; manual_now = '2016-08-01 01:02:03'; num_wkrs = 8; load_all_templates = 'y'; cleanup_interval = 50; hzip_enabled = 'y'; hdiff_enabled ='y'; manual_now = '2016-08-01 01:02:03';
load_all_imglinks = 'y';
// uncomment the following 3 lines if using the build script as a "worker" helping a "server" // uncomment the following 3 lines if using the build script as a "worker" helping a "server"
// num_pages_in_pool = 32000; // num_pages_in_pool = 32000;
@ -443,8 +450,7 @@ app.bldr.cmds {
// note that if multi-machine mode is enabled, all worker directories must be manually copied to the server directory (a build command will be added later) // note that if multi-machine mode is enabled, all worker directories must be manually copied to the server directory (a build command will be added later)
add ('simple.wikipedia.org' , 'wiki.mass_parse.make'); add ('simple.wikipedia.org' , 'wiki.mass_parse.make');
*/
// aggregate the lnkis // aggregate the lnkis
add ('simple.wikipedia.org' , 'file.lnki_regy'); add ('simple.wikipedia.org' , 'file.lnki_regy');
@ -492,7 +498,10 @@ app.bldr.run;
<pre class='code'> <pre class='code'>
app.bldr.pause_at_end_('n'); app.bldr.pause_at_end_('n');
app.scripts.run_file_by_type('xowa_cfg_app'); app.scripts.run_file_by_type('xowa_cfg_app');
app.cfgs.get('app.user.cfg.security.web_access_enabled', 'app').val = 'y'; app.cfg.set_temp('app', 'xowa.app.web.enabled', 'y');
app.cfg.set_temp('app', 'xowa.bldr.db.layout_size.text', '0');
app.cfg.set_temp('app', 'xowa.bldr.db.layout_size.html', '0');
app.cfg.set_temp('app', 'xowa.bldr.db.layout_size.file', '0');
app.bldr.cmds { app.bldr.cmds {
/* /*
add ('www.wikidata.org' , 'util.cleanup') {delete_all = 'y';} add ('www.wikidata.org' , 'util.cleanup') {delete_all = 'y';}
@ -502,56 +511,47 @@ app.bldr.cmds {
add ('www.wikidata.org' , 'util.download') {dump_type = 'image';} add ('www.wikidata.org' , 'util.download') {dump_type = 'image';}
add ('www.wikidata.org' , 'text.init'); add ('www.wikidata.org' , 'text.init');
add ('www.wikidata.org' , 'text.page'); add ('www.wikidata.org' , 'text.page');
add ('www.wikidata.org' , 'text.cat.core');
add ('www.wikidata.org' , 'text.cat.link');
add ('www.wikidata.org' , 'text.cat.hidden');
add ('www.wikidata.org' , 'text.term'); add ('www.wikidata.org' , 'text.term');
add ('www.wikidata.org' , 'text.css'); add ('www.wikidata.org' , 'text.css');
add ('www.wikidata.org' , 'wiki.image'); add ('www.wikidata.org' , 'wiki.image');
add ('www.wikidata.org' , 'file.page_regy') {build_commons = 'y'}
add ('www.wikidata.org' , 'wiki.page_dump.make'); add ('www.wikidata.org' , 'wiki.page_dump.make');
add ('www.wikidata.org' , 'wiki.page_props');
add ('www.wikidata.org' , 'wiki.categorylinks');
add ('www.wikidata.org' , 'wiki.redirect') {commit_interval = 1000; progress_interval = 100; cleanup_interval = 100;} add ('www.wikidata.org' , 'wiki.redirect') {commit_interval = 1000; progress_interval = 100; cleanup_interval = 100;}
add ('www.wikidata.org' , 'util.cleanup') {delete_tmp = 'y'; delete_by_match('*.xml|*.sql|*.bz2|*.gz');} // add ('www.wikidata.org' , 'util.cleanup') {delete_tmp = 'y'; delete_by_match('*.xml|*.sql|*.bz2|*.gz');}
add ('commons.wikimedia.org' , 'util.cleanup') {delete_all = 'y';} add ('commons.wikimedia.org' , 'util.cleanup') {delete_all = 'y';}
add ('commons.wikimedia.org' , 'util.download') {dump_type = 'pages-articles';} add ('commons.wikimedia.org' , 'util.download') {dump_type = 'pages-articles';}
add ('commons.wikimedia.org' , 'util.download') {dump_type = 'image';}
add ('commons.wikimedia.org' , 'util.download') {dump_type = 'categorylinks';} add ('commons.wikimedia.org' , 'util.download') {dump_type = 'categorylinks';}
add ('commons.wikimedia.org' , 'util.download') {dump_type = 'page_props';} add ('commons.wikimedia.org' , 'util.download') {dump_type = 'page_props';}
add ('commons.wikimedia.org' , 'util.download') {dump_type = 'image';}
add ('commons.wikimedia.org' , 'text.init'); add ('commons.wikimedia.org' , 'text.init');
add ('commons.wikimedia.org' , 'text.page'); add ('commons.wikimedia.org' , 'text.page');
add ('commons.wikimedia.org' , 'text.cat.core');
add ('commons.wikimedia.org' , 'text.cat.link');
add ('commons.wikimedia.org' , 'text.cat.hidden');
add ('commons.wikimedia.org' , 'text.term'); add ('commons.wikimedia.org' , 'text.term');
add ('commons.wikimedia.org' , 'text.css'); add ('commons.wikimedia.org' , 'text.css');
add ('commons.wikimedia.org' , 'wiki.image'); add ('commons.wikimedia.org' , 'wiki.image');
add ('commons.wikimedia.org' , 'file.page_regy') {build_commons = 'y'} add ('commons.wikimedia.org' , 'file.page_regy') {build_commons = 'y'}
add ('commons.wikimedia.org' , 'wiki.page_dump.make'); add ('commons.wikimedia.org' , 'wiki.page_dump.make');
add ('commons.wikimedia.org' , 'wiki.redirect') {commit_interval = 1000; progress_interval = 100; cleanup_interval = 100;} add ('commons.wikimedia.org' , 'wiki.redirect') {commit_interval = 1000; progress_interval = 100; cleanup_interval = 100;}
*/ // add ('commons.wikimedia.org' , 'util.cleanup') {delete_tmp = 'y'; delete_by_match('*.xml|*.sql|*.bz2|*.gz');}
/*
// en.wikipedia.org
add ('en.wikipedia.org' , 'util.download') {dump_type = 'pages-articles';} add ('en.wikipedia.org' , 'util.download') {dump_type = 'pages-articles';}
add ('en.wikipedia.org' , 'util.download') {dump_type = 'pagelinks';}
add ('en.wikipedia.org' , 'util.download') {dump_type = 'categorylinks';} add ('en.wikipedia.org' , 'util.download') {dump_type = 'categorylinks';}
add ('en.wikipedia.org' , 'util.download') {dump_type = 'page_props';} add ('en.wikipedia.org' , 'util.download') {dump_type = 'page_props';}
add ('en.wikipedia.org' , 'util.download') {dump_type = 'image';} add ('en.wikipedia.org' , 'util.download') {dump_type = 'image';}
add ('en.wikipedia.org' , 'util.download') {dump_type = 'pagelinks';}
add ('en.wikipedia.org' , 'util.download') {dump_type = 'imagelinks';} add ('en.wikipedia.org' , 'util.download') {dump_type = 'imagelinks';}
*/ */
/* /*
// en.wikipedia.org
add ('en.wikipedia.org' , 'text.init'); add ('en.wikipedia.org' , 'text.init');
add ('en.wikipedia.org' , 'text.page') {redirect_id_enabled = 'y';} add ('en.wikipedia.org' , 'text.page') {redirect_id_enabled = 'y';}
add ('en.wikipedia.org' , 'text.search'); add ('en.wikipedia.org' , 'text.search');
add ('en.wikipedia.org' , 'text.css'); add ('en.wikipedia.org' , 'text.css');
add ('en.wikipedia.org' , 'text.cat.core');
add ('en.wikipedia.org' , 'text.cat.link');
add ('en.wikipedia.org' , 'text.cat.hidden');
add ('en.wikipedia.org' , 'text.term'); add ('en.wikipedia.org' , 'text.term');
// add ('en.wikipedia.org' , 'wiki.redirect') {commit_interval = 1000; progress_interval = 100; cleanup_interval = 100;}
add ('en.wikipedia.org' , 'wiki.image'); add ('en.wikipedia.org' , 'wiki.image');
add ('en.wikipedia.org' , 'wiki.page_dump.make');
add ('en.wikipedia.org' , 'wiki.page_link');
add ('en.wikipedia.org' , 'wiki.imagelinks'); add ('en.wikipedia.org' , 'wiki.imagelinks');
add ('en.wikipedia.org' , 'wiki.page_dump.make');
add ('en.wikipedia.org' , 'wiki.redirect') {commit_interval = 1000; progress_interval = 100; cleanup_interval = 100;}
add ('en.wikipedia.org' , 'wiki.page_link');
add ('en.wikipedia.org' , 'search.page__page_score') {iteration_max = 100;} add ('en.wikipedia.org' , 'search.page__page_score') {iteration_max = 100;}
add ('en.wikipedia.org' , 'search.link__link_score') {page_rank_enabled = 'y'; add ('en.wikipedia.org' , 'search.link__link_score') {page_rank_enabled = 'y';
score_adjustment_mgr { score_adjustment_mgr {
@ -566,32 +566,43 @@ app.bldr.cmds {
} }
} }
add ('en.wikipedia.org' , 'search.word__link_count') add ('en.wikipedia.org' , 'search.word__link_count')
/* add ('en.wikipedia.org' , 'wiki.page_props');
// SELECT * FROM xowa_cfg WHERE cfg_key = 'props.modified_latest'; add ('en.wikipedia.org' , 'wiki.categorylinks');
add ('en.wikipedia.org' , 'file.lnki_temp') {
commit_interval = 10000; progress_interval = 50; cleanup_interval = 50; select_size = 25;
ns_ids = '0|4|14|100';
hdump_bldr {enabled = 'y'; hzip_enabled = 'y'; hzip_diff = 'y';}
}
add ('en.wikipedia.org' , 'file.lnki_regy');
add ('commons.wikimedia.org' , 'file.page_regy') {build_commons = 'y'}
add ('en.wikipedia.org' , 'file.page_regy') {build_commons = 'n';}
add ('en.wikipedia.org' , 'wiki.image');
add ('en.wikipedia.org' , 'file.orig_regy');
// SELECT * FROM orig_regy WHERE lnki_ttl = 'BSicon_CONTr.svg';
// SELECT * FROM page_regy WHERE src_ttl = 'BSicon_CONTr.svg';
add ('en.wikipedia.org' , 'file.xfer_temp.thumb');
// SELECT Count(*) FROM xfer_regy WHERE xfer_status = 0;
// SELECT * FROM xfer_regy WHERE xfer_status = 0 AND lnki_page_id = 372692; --en.w:Featured_picture_candidates
add ('en.wikipedia.org' , 'file.xfer_regy');
*/ */
/* /*
add ('en.wikipedia.org' , 'file.page_regy') {build_commons = 'n'}
add ('en.wikipedia.org' , 'wiki.mass_parse.init') {cfg {ns_ids = '0|4|100|14|8';}}
// add ('en.wikipedia.org' , 'wiki.mass_parse.resume');
add ('en.wikipedia.org' , 'wiki.mass_parse.exec') {cfg {
num_wkrs = 8; load_all_templates = 'y'; load_ifexists_ns = '*'; cleanup_interval = 25; hzip_enabled = 'y'; hdiff_enabled ='y'; manual_now = '2017-01-01 01:02:03';}
// num_wkrs = 1; load_all_templates = 'n'; load_all_imglnks = 'n'; cleanup_interval = 50; hzip_enabled = 'y'; hdiff_enabled ='y'; manual_now = '2016-07-28 01:02:03';}
}
add ('en.wikipedia.org' , 'wiki.mass_parse.make');
*/
/*
add ('en.wikipedia.org' , 'file.lnki_temp') {
commit_interval = 10000; progress_interval = 50; cleanup_interval = 50; select_size = 25;
ns_ids = '0|4|14|100|12|8|6|10|828|108|118|446|710|2300|2302|2600';
hdump_bldr {enabled = 'y'; hzip_enabled = 'y'; hzip_diff = 'y';}
}
*/
/*
add ('commons.wikimedia.org' , 'file.page_regy') {build_commons = 'y'}
add ('en.wikipedia.org' , 'file.page_regy') {build_commons = 'n';}
add ('en.wikipedia.org' , 'file.lnki_regy');
// add ('en.wikipedia.org' , 'wiki.image');
add ('en.wikipedia.org' , 'file.orig_regy');
add ('en.wikipedia.org' , 'file.xfer_temp.thumb');
add ('en.wikipedia.org' , 'file.xfer_regy');
add ('en.wikipedia.org' , 'file.xfer_regy_update'); add ('en.wikipedia.org' , 'file.xfer_regy_update');
*/
/*
add ('en.wikipedia.org' , 'file.fsdb_make') { add ('en.wikipedia.org' , 'file.fsdb_make') {
commit_interval = 1000; progress_interval = 200; select_interval = 10000; commit_interval = 1000; progress_interval = 200; select_interval = 10000;
ns_ids = '0|4|14|100'; ns_ids = '0|4|100|14|8';
// // specify whether original wiki databases are v1 (.sqlite3) or v2 (.xowa) // // specify whether original wiki databases are v1 (.sqlite3) or v2 (.xowa)
// // src_bin_mgr__fsdb_version = 'v2'; // src_bin_mgr__fsdb_version = 'v2';
// trg_bin_mgr__fsdb_version = 'v1'; // trg_bin_mgr__fsdb_version = 'v1';
@ -603,6 +614,7 @@ app.bldr.cmds {
} }
add ('en.wikipedia.org' , 'file.orig_reg'); add ('en.wikipedia.org' , 'file.orig_reg');
add ('en.wikipedia.org' , 'wiki.page_dump.drop'); add ('en.wikipedia.org' , 'wiki.page_dump.drop');
add ('en.wikipedia.org' , 'file.page_file_map.create');
*/ */
} }
app.bldr.run; app.bldr.run;
@ -614,6 +626,9 @@ app.bldr.run;
<li> <li>
2016-10-12: explicitly set web_access_enabled to y 2016-10-12: explicitly set web_access_enabled to y
</li> </li>
<li>
2017-02-02: updated script for multi-threaded version and new options
</li>
</ul> </ul>
</div> </div>

109
home/wiki/Dev/Command-line/Thumbs.html
View File

@ -68,7 +68,7 @@
<a href="#Requirements"><span class="tocnumber">2</span> <span class="toctext">Requirements</span></a> <a href="#Requirements"><span class="tocnumber">2</span> <span class="toctext">Requirements</span></a>
<ul> <ul>
<li class="toclevel-2 tocsection-3"> <li class="toclevel-2 tocsection-3">
<a href="#commons.wikimedia.org_.28thum"><span class="tocnumber">2.1</span> <span class="toctext">commons.wikimedia.org (thum</span></a> <a href="#commons.wikimedia.org"><span class="tocnumber">2.1</span> <span class="toctext">commons.wikimedia.org</span></a>
</li> </li>
<li class="toclevel-2 tocsection-4"> <li class="toclevel-2 tocsection-4">
<a href="#www.wikidata.org"><span class="tocnumber">2.2</span> <span class="toctext">www.wikidata.org</span></a> <a href="#www.wikidata.org"><span class="tocnumber">2.2</span> <span class="toctext">www.wikidata.org</span></a>
@ -158,7 +158,7 @@
<span class="mw-headline" id="Requirements">Requirements</span> <span class="mw-headline" id="Requirements">Requirements</span>
</h2> </h2>
<h3> <h3>
<span class="mw-headline" id="commons.wikimedia.org_.28thum">commons.wikimedia.org (thum</span> <span class="mw-headline" id="commons.wikimedia.org">commons.wikimedia.org</span>
</h3> </h3>
<p> <p>
You will need the latest version of commons.wikimedia.org. Note that if you have an older version, you will have missing images or wrong size information. You will need the latest version of commons.wikimedia.org. Note that if you have an older version, you will have missing images or wrong size information.
@ -304,6 +304,10 @@
<pre class='code'> <pre class='code'>
app.bldr.pause_at_end_('n'); app.bldr.pause_at_end_('n');
app.scripts.run_file_by_type('xowa_cfg_app'); app.scripts.run_file_by_type('xowa_cfg_app');
app.cfg.set_temp('app', 'xowa.app.web.enabled', 'y');
app.cfg.set_temp('app', 'xowa.bldr.db.layout_size.text', '0');
app.cfg.set_temp('app', 'xowa.bldr.db.layout_size.html', '0');
app.cfg.set_temp('app', 'xowa.bldr.db.layout_size.file', '0');
app.bldr.cmds { app.bldr.cmds {
// build commons database; this only needs to be done once, whenever commons is updated // build commons database; this only needs to be done once, whenever commons is updated
add ('commons.wikimedia.org' , 'util.cleanup') {delete_all = 'y';} add ('commons.wikimedia.org' , 'util.cleanup') {delete_all = 'y';}
@ -391,9 +395,11 @@ app.bldr.cmds {
// cleanup all downloaded files as well as temporary files // cleanup all downloaded files as well as temporary files
add ('simple.wikipedia.org' , 'util.cleanup') {delete_tmp = 'y'; delete_by_match('*.xml|*.sql|*.bz2|*.gz');} add ('simple.wikipedia.org' , 'util.cleanup') {delete_tmp = 'y'; delete_by_match('*.xml|*.sql|*.bz2|*.gz');}
// OBSOLETE: use v2
// v1 html generator // v1 html generator
// parse every page in the listed namespace and gather data on their lnkis. // parse every page in the listed namespace and gather data on their lnkis.
// this step will take the longest amount of time. // this step will take the longest amount of time.
/*
add ('simple.wikipedia.org' , 'file.lnki_temp') { add ('simple.wikipedia.org' , 'file.lnki_temp') {
// save data every # of pages // save data every # of pages
commit_interval = 10000; commit_interval = 10000;
@ -426,13 +432,14 @@ app.bldr.cmds {
hzip_diff = 'y'; hzip_diff = 'y';
} }
} }
*/
// v2 html generator; allows for multi-threaded / multi-machine builds // v2 html generator; allows for multi-threaded / multi-machine builds
/* add ('simple.wikipedia.org' , 'wiki.mass_parse.init') {cfg {ns_ids = '0|4|14|8';}}
add ('simple.wikipedia.org' , 'wiki.mass_parse.init') {cfg {ns_ids = '0|4|14';}}
add ('simple.wikipedia.org' , 'wiki.mass_parse.exec') { add ('simple.wikipedia.org' , 'wiki.mass_parse.exec') {
cfg { cfg {
num_wkrs = 8; load_all_templates = 'y'; cleanup_interval = 50; hzip_enabled = 'y'; hdiff_enabled ='y'; manual_now = '2016-08-01 01:02:03'; num_wkrs = 8; load_all_templates = 'y'; cleanup_interval = 50; hzip_enabled = 'y'; hdiff_enabled ='y'; manual_now = '2016-08-01 01:02:03';
load_all_imglinks = 'y';
// uncomment the following 3 lines if using the build script as a "worker" helping a "server" // uncomment the following 3 lines if using the build script as a "worker" helping a "server"
// num_pages_in_pool = 32000; // num_pages_in_pool = 32000;
@ -443,8 +450,7 @@ app.bldr.cmds {
// note that if multi-machine mode is enabled, all worker directories must be manually copied to the server directory (a build command will be added later) // note that if multi-machine mode is enabled, all worker directories must be manually copied to the server directory (a build command will be added later)
add ('simple.wikipedia.org' , 'wiki.mass_parse.make'); add ('simple.wikipedia.org' , 'wiki.mass_parse.make');
*/
// aggregate the lnkis // aggregate the lnkis
add ('simple.wikipedia.org' , 'file.lnki_regy'); add ('simple.wikipedia.org' , 'file.lnki_regy');
@ -492,7 +498,10 @@ app.bldr.run;
<pre class='code'> <pre class='code'>
app.bldr.pause_at_end_('n'); app.bldr.pause_at_end_('n');
app.scripts.run_file_by_type('xowa_cfg_app'); app.scripts.run_file_by_type('xowa_cfg_app');
app.cfgs.get('app.user.cfg.security.web_access_enabled', 'app').val = 'y'; app.cfg.set_temp('app', 'xowa.app.web.enabled', 'y');
app.cfg.set_temp('app', 'xowa.bldr.db.layout_size.text', '0');
app.cfg.set_temp('app', 'xowa.bldr.db.layout_size.html', '0');
app.cfg.set_temp('app', 'xowa.bldr.db.layout_size.file', '0');
app.bldr.cmds { app.bldr.cmds {
/* /*
add ('www.wikidata.org' , 'util.cleanup') {delete_all = 'y';} add ('www.wikidata.org' , 'util.cleanup') {delete_all = 'y';}
@ -502,56 +511,47 @@ app.bldr.cmds {
add ('www.wikidata.org' , 'util.download') {dump_type = 'image';} add ('www.wikidata.org' , 'util.download') {dump_type = 'image';}
add ('www.wikidata.org' , 'text.init'); add ('www.wikidata.org' , 'text.init');
add ('www.wikidata.org' , 'text.page'); add ('www.wikidata.org' , 'text.page');
add ('www.wikidata.org' , 'text.cat.core');
add ('www.wikidata.org' , 'text.cat.link');
add ('www.wikidata.org' , 'text.cat.hidden');
add ('www.wikidata.org' , 'text.term'); add ('www.wikidata.org' , 'text.term');
add ('www.wikidata.org' , 'text.css'); add ('www.wikidata.org' , 'text.css');
add ('www.wikidata.org' , 'wiki.image'); add ('www.wikidata.org' , 'wiki.image');
add ('www.wikidata.org' , 'file.page_regy') {build_commons = 'y'}
add ('www.wikidata.org' , 'wiki.page_dump.make'); add ('www.wikidata.org' , 'wiki.page_dump.make');
add ('www.wikidata.org' , 'wiki.page_props');
add ('www.wikidata.org' , 'wiki.categorylinks');
add ('www.wikidata.org' , 'wiki.redirect') {commit_interval = 1000; progress_interval = 100; cleanup_interval = 100;} add ('www.wikidata.org' , 'wiki.redirect') {commit_interval = 1000; progress_interval = 100; cleanup_interval = 100;}
add ('www.wikidata.org' , 'util.cleanup') {delete_tmp = 'y'; delete_by_match('*.xml|*.sql|*.bz2|*.gz');} // add ('www.wikidata.org' , 'util.cleanup') {delete_tmp = 'y'; delete_by_match('*.xml|*.sql|*.bz2|*.gz');}
add ('commons.wikimedia.org' , 'util.cleanup') {delete_all = 'y';} add ('commons.wikimedia.org' , 'util.cleanup') {delete_all = 'y';}
add ('commons.wikimedia.org' , 'util.download') {dump_type = 'pages-articles';} add ('commons.wikimedia.org' , 'util.download') {dump_type = 'pages-articles';}
add ('commons.wikimedia.org' , 'util.download') {dump_type = 'image';}
add ('commons.wikimedia.org' , 'util.download') {dump_type = 'categorylinks';} add ('commons.wikimedia.org' , 'util.download') {dump_type = 'categorylinks';}
add ('commons.wikimedia.org' , 'util.download') {dump_type = 'page_props';} add ('commons.wikimedia.org' , 'util.download') {dump_type = 'page_props';}
add ('commons.wikimedia.org' , 'util.download') {dump_type = 'image';}
add ('commons.wikimedia.org' , 'text.init'); add ('commons.wikimedia.org' , 'text.init');
add ('commons.wikimedia.org' , 'text.page'); add ('commons.wikimedia.org' , 'text.page');
add ('commons.wikimedia.org' , 'text.cat.core');
add ('commons.wikimedia.org' , 'text.cat.link');
add ('commons.wikimedia.org' , 'text.cat.hidden');
add ('commons.wikimedia.org' , 'text.term'); add ('commons.wikimedia.org' , 'text.term');
add ('commons.wikimedia.org' , 'text.css'); add ('commons.wikimedia.org' , 'text.css');
add ('commons.wikimedia.org' , 'wiki.image'); add ('commons.wikimedia.org' , 'wiki.image');
add ('commons.wikimedia.org' , 'file.page_regy') {build_commons = 'y'} add ('commons.wikimedia.org' , 'file.page_regy') {build_commons = 'y'}
add ('commons.wikimedia.org' , 'wiki.page_dump.make'); add ('commons.wikimedia.org' , 'wiki.page_dump.make');
add ('commons.wikimedia.org' , 'wiki.redirect') {commit_interval = 1000; progress_interval = 100; cleanup_interval = 100;} add ('commons.wikimedia.org' , 'wiki.redirect') {commit_interval = 1000; progress_interval = 100; cleanup_interval = 100;}
*/ // add ('commons.wikimedia.org' , 'util.cleanup') {delete_tmp = 'y'; delete_by_match('*.xml|*.sql|*.bz2|*.gz');}
/*
// en.wikipedia.org
add ('en.wikipedia.org' , 'util.download') {dump_type = 'pages-articles';} add ('en.wikipedia.org' , 'util.download') {dump_type = 'pages-articles';}
add ('en.wikipedia.org' , 'util.download') {dump_type = 'pagelinks';}
add ('en.wikipedia.org' , 'util.download') {dump_type = 'categorylinks';} add ('en.wikipedia.org' , 'util.download') {dump_type = 'categorylinks';}
add ('en.wikipedia.org' , 'util.download') {dump_type = 'page_props';} add ('en.wikipedia.org' , 'util.download') {dump_type = 'page_props';}
add ('en.wikipedia.org' , 'util.download') {dump_type = 'image';} add ('en.wikipedia.org' , 'util.download') {dump_type = 'image';}
add ('en.wikipedia.org' , 'util.download') {dump_type = 'pagelinks';}
add ('en.wikipedia.org' , 'util.download') {dump_type = 'imagelinks';} add ('en.wikipedia.org' , 'util.download') {dump_type = 'imagelinks';}
*/ */
/* /*
// en.wikipedia.org
add ('en.wikipedia.org' , 'text.init'); add ('en.wikipedia.org' , 'text.init');
add ('en.wikipedia.org' , 'text.page') {redirect_id_enabled = 'y';} add ('en.wikipedia.org' , 'text.page') {redirect_id_enabled = 'y';}
add ('en.wikipedia.org' , 'text.search'); add ('en.wikipedia.org' , 'text.search');
add ('en.wikipedia.org' , 'text.css'); add ('en.wikipedia.org' , 'text.css');
add ('en.wikipedia.org' , 'text.cat.core');
add ('en.wikipedia.org' , 'text.cat.link');
add ('en.wikipedia.org' , 'text.cat.hidden');
add ('en.wikipedia.org' , 'text.term'); add ('en.wikipedia.org' , 'text.term');
// add ('en.wikipedia.org' , 'wiki.redirect') {commit_interval = 1000; progress_interval = 100; cleanup_interval = 100;}
add ('en.wikipedia.org' , 'wiki.image'); add ('en.wikipedia.org' , 'wiki.image');
add ('en.wikipedia.org' , 'wiki.page_dump.make');
add ('en.wikipedia.org' , 'wiki.page_link');
add ('en.wikipedia.org' , 'wiki.imagelinks'); add ('en.wikipedia.org' , 'wiki.imagelinks');
add ('en.wikipedia.org' , 'wiki.page_dump.make');
add ('en.wikipedia.org' , 'wiki.redirect') {commit_interval = 1000; progress_interval = 100; cleanup_interval = 100;}
add ('en.wikipedia.org' , 'wiki.page_link');
add ('en.wikipedia.org' , 'search.page__page_score') {iteration_max = 100;} add ('en.wikipedia.org' , 'search.page__page_score') {iteration_max = 100;}
add ('en.wikipedia.org' , 'search.link__link_score') {page_rank_enabled = 'y'; add ('en.wikipedia.org' , 'search.link__link_score') {page_rank_enabled = 'y';
score_adjustment_mgr { score_adjustment_mgr {
@ -566,32 +566,43 @@ app.bldr.cmds {
} }
} }
add ('en.wikipedia.org' , 'search.word__link_count') add ('en.wikipedia.org' , 'search.word__link_count')
/* add ('en.wikipedia.org' , 'wiki.page_props');
// SELECT * FROM xowa_cfg WHERE cfg_key = 'props.modified_latest'; add ('en.wikipedia.org' , 'wiki.categorylinks');
add ('en.wikipedia.org' , 'file.lnki_temp') {
commit_interval = 10000; progress_interval = 50; cleanup_interval = 50; select_size = 25;
ns_ids = '0|4|14|100';
hdump_bldr {enabled = 'y'; hzip_enabled = 'y'; hzip_diff = 'y';}
}
add ('en.wikipedia.org' , 'file.lnki_regy');
add ('commons.wikimedia.org' , 'file.page_regy') {build_commons = 'y'}
add ('en.wikipedia.org' , 'file.page_regy') {build_commons = 'n';}
add ('en.wikipedia.org' , 'wiki.image');
add ('en.wikipedia.org' , 'file.orig_regy');
// SELECT * FROM orig_regy WHERE lnki_ttl = 'BSicon_CONTr.svg';
// SELECT * FROM page_regy WHERE src_ttl = 'BSicon_CONTr.svg';
add ('en.wikipedia.org' , 'file.xfer_temp.thumb');
// SELECT Count(*) FROM xfer_regy WHERE xfer_status = 0;
// SELECT * FROM xfer_regy WHERE xfer_status = 0 AND lnki_page_id = 372692; --en.w:Featured_picture_candidates
add ('en.wikipedia.org' , 'file.xfer_regy');
*/ */
/* /*
add ('en.wikipedia.org' , 'file.page_regy') {build_commons = 'n'}
add ('en.wikipedia.org' , 'wiki.mass_parse.init') {cfg {ns_ids = '0|4|100|14|8';}}
// add ('en.wikipedia.org' , 'wiki.mass_parse.resume');
add ('en.wikipedia.org' , 'wiki.mass_parse.exec') {cfg {
num_wkrs = 8; load_all_templates = 'y'; load_ifexists_ns = '*'; cleanup_interval = 25; hzip_enabled = 'y'; hdiff_enabled ='y'; manual_now = '2017-01-01 01:02:03';}
// num_wkrs = 1; load_all_templates = 'n'; load_all_imglnks = 'n'; cleanup_interval = 50; hzip_enabled = 'y'; hdiff_enabled ='y'; manual_now = '2016-07-28 01:02:03';}
}
add ('en.wikipedia.org' , 'wiki.mass_parse.make');
*/
/*
add ('en.wikipedia.org' , 'file.lnki_temp') {
commit_interval = 10000; progress_interval = 50; cleanup_interval = 50; select_size = 25;
ns_ids = '0|4|14|100|12|8|6|10|828|108|118|446|710|2300|2302|2600';
hdump_bldr {enabled = 'y'; hzip_enabled = 'y'; hzip_diff = 'y';}
}
*/
/*
add ('commons.wikimedia.org' , 'file.page_regy') {build_commons = 'y'}
add ('en.wikipedia.org' , 'file.page_regy') {build_commons = 'n';}
add ('en.wikipedia.org' , 'file.lnki_regy');
// add ('en.wikipedia.org' , 'wiki.image');
add ('en.wikipedia.org' , 'file.orig_regy');
add ('en.wikipedia.org' , 'file.xfer_temp.thumb');
add ('en.wikipedia.org' , 'file.xfer_regy');
add ('en.wikipedia.org' , 'file.xfer_regy_update'); add ('en.wikipedia.org' , 'file.xfer_regy_update');
*/
/*
add ('en.wikipedia.org' , 'file.fsdb_make') { add ('en.wikipedia.org' , 'file.fsdb_make') {
commit_interval = 1000; progress_interval = 200; select_interval = 10000; commit_interval = 1000; progress_interval = 200; select_interval = 10000;
ns_ids = '0|4|14|100'; ns_ids = '0|4|100|14|8';
// // specify whether original wiki databases are v1 (.sqlite3) or v2 (.xowa) // // specify whether original wiki databases are v1 (.sqlite3) or v2 (.xowa)
// // src_bin_mgr__fsdb_version = 'v2'; // src_bin_mgr__fsdb_version = 'v2';
// trg_bin_mgr__fsdb_version = 'v1'; // trg_bin_mgr__fsdb_version = 'v1';
@ -603,6 +614,7 @@ app.bldr.cmds {
} }
add ('en.wikipedia.org' , 'file.orig_reg'); add ('en.wikipedia.org' , 'file.orig_reg');
add ('en.wikipedia.org' , 'wiki.page_dump.drop'); add ('en.wikipedia.org' , 'wiki.page_dump.drop');
add ('en.wikipedia.org' , 'file.page_file_map.create');
*/ */
} }
app.bldr.run; app.bldr.run;
@ -614,6 +626,9 @@ app.bldr.run;
<li> <li>
2016-10-12: explicitly set web_access_enabled to y 2016-10-12: explicitly set web_access_enabled to y
</li> </li>
<li>
2017-02-02: updated script for multi-threaded version and new options
</li>
</ul> </ul>
</div> </div>

2
home/wiki/Help/Contents.html
View File

@ -181,7 +181,7 @@
<a href="http://xowa.org/home/wiki/Dev/Command-line.html" id="xolnki_24" title="Dev/Command-line" class="xowa-visited">Overview</a> <a href="http://xowa.org/home/wiki/Dev/Command-line.html" id="xolnki_24" title="Dev/Command-line" class="xowa-visited">Overview</a>
</li> </li>
<li> <li>
<a href="http://xowa.org/home/wiki/Dev/Command-line/Dumps.html" id="xolnki_25" title="Dev/Command-line/Dumps">Image dumps</a> <a href="http://xowa.org/home/wiki/Dev/Command-line/Dumps.html" id="xolnki_25" title="Dev/Command-line/Dumps" class="xowa-visited">Image dumps</a>
</li> </li>
<li> <li>
<a href="http://xowa.org/home/wiki/Dev/Command-line/Wikidata.html" id="xolnki_26" title="Dev/Command-line/Wikidata">Wikidata</a> <a href="http://xowa.org/home/wiki/Dev/Command-line/Wikidata.html" id="xolnki_26" title="Dev/Command-line/Wikidata">Wikidata</a>

12
home/wiki/Options.html
View File

@ -278,17 +278,17 @@
</div> </div>
<div id='xowa.files.cache.info__undo' data-xocfg-type='memo' class='xocfg_itm_delete xocfg_itm_hide'> <div id='xowa.files.cache.info__undo' data-xocfg-type='memo' class='xocfg_itm_delete xocfg_itm_hide'>
<span class='xoimg_btn_x16 xoimg_list_undo' onclick='return xo.cfg_edit.delete__send("xowa.files.cache.info")' title="reset to &quot;cache folder: C:\xowa\file <span class='xoimg_btn_x16 xoimg_list_undo' onclick='return xo.cfg_edit.delete__send("xowa.files.cache.info")' title="reset to &quot;cache folder: C:\xowa\file
space used: 0.000 B space used: 69.812 KB
file count: 0 file count: 1
oldest file: oldest file: 2017-02-02 11:03:45
&quot;">&nbsp;</span> &quot;">&nbsp;</span>
</div> </div>
<div class='xocfg_itm_data'> <div class='xocfg_itm_data'>
<textarea id="xowa.files.cache.info" data-xocfg-key="xowa.files.cache.info" data-xocfg-type="memo" accesskey="d" class="xocfg_data__memo xocfg_data__readonly" readonly="true"> <textarea id="xowa.files.cache.info" data-xocfg-key="xowa.files.cache.info" data-xocfg-type="memo" accesskey="d" class="xocfg_data__memo xocfg_data__readonly" readonly="true">
cache folder: C:\xowa\file cache folder: C:\xowa\file
space used: 0.000 B space used: 69.812 KB
file count: 0 file count: 1
oldest file: oldest file: 2017-02-02 11:03:45
</textarea> </textarea>
</div><input type='hidden' id='xowa.files.cache.info__key_box' value='xowa.files.cache.info'> <input type='hidden' id='xowa.files.cache.info__ctx_box' value='app'> </div><input type='hidden' id='xowa.files.cache.info__key_box' value='xowa.files.cache.info'> <input type='hidden' id='xowa.files.cache.info__ctx_box' value='app'>
</div> </div>

2
home/wiki/Options/Bookmarks.html
View File

@ -264,7 +264,7 @@ li.active a, li.active a:hover
</div> </div>
</div> </div>
<p> <p>
gplx.Gfo_invk__noop@7c584ba gplx.Gfo_invk__noop@70b02e04
</p> </p>
</div> </div>
</div> </div>

3
home/wiki/Options/Dev.html
View File

@ -312,8 +312,7 @@
<div class='xocfg_itm_data'> <div class='xocfg_itm_data'>
<textarea id="xowa.app.startup.tabs.previous_list" data-xocfg-key="xowa.app.startup.tabs.previous_list" data-xocfg-type="memo" accesskey="d" class="xocfg_data__memo xocfg_data__readonly" readonly="true"> <textarea id="xowa.app.startup.tabs.previous_list" data-xocfg-key="xowa.app.startup.tabs.previous_list" data-xocfg-type="memo" accesskey="d" class="xocfg_data__memo xocfg_data__readonly" readonly="true">
home/wiki/Special:XowaCfg home/wiki/Special:XowaCfg
home/wiki/App/Wiki_types/Wikia.com home/wiki/Dev/Command-line/Dumps
home/wiki/Dev/File/Setup/Windows
</textarea> </textarea>
</div><input type='hidden' id='xowa.app.startup.tabs.previous_list__key_box' value='xowa.app.startup.tabs.previous_list'> <input type='hidden' id='xowa.app.startup.tabs.previous_list__ctx_box' value='app'> </div><input type='hidden' id='xowa.app.startup.tabs.previous_list__key_box' value='xowa.app.startup.tabs.previous_list'> <input type='hidden' id='xowa.app.startup.tabs.previous_list__ctx_box' value='app'>
</div> </div>

12
home/wiki/Options/Files.html
View File

@ -278,17 +278,17 @@
</div> </div>
<div id='xowa.files.cache.info__undo' data-xocfg-type='memo' class='xocfg_itm_delete xocfg_itm_hide'> <div id='xowa.files.cache.info__undo' data-xocfg-type='memo' class='xocfg_itm_delete xocfg_itm_hide'>
<span class='xoimg_btn_x16 xoimg_list_undo' onclick='return xo.cfg_edit.delete__send("xowa.files.cache.info")' title="reset to &quot;cache folder: C:\xowa\file <span class='xoimg_btn_x16 xoimg_list_undo' onclick='return xo.cfg_edit.delete__send("xowa.files.cache.info")' title="reset to &quot;cache folder: C:\xowa\file
space used: 0.000 B space used: 69.812 KB
file count: 0 file count: 1
oldest file: oldest file: 2017-02-02 11:03:45
&quot;">&nbsp;</span> &quot;">&nbsp;</span>
</div> </div>
<div class='xocfg_itm_data'> <div class='xocfg_itm_data'>
<textarea id="xowa.files.cache.info" data-xocfg-key="xowa.files.cache.info" data-xocfg-type="memo" accesskey="d" class="xocfg_data__memo xocfg_data__readonly" readonly="true"> <textarea id="xowa.files.cache.info" data-xocfg-key="xowa.files.cache.info" data-xocfg-type="memo" accesskey="d" class="xocfg_data__memo xocfg_data__readonly" readonly="true">
cache folder: C:\xowa\file cache folder: C:\xowa\file
space used: 0.000 B space used: 69.812 KB
file count: 0 file count: 1
oldest file: oldest file: 2017-02-02 11:03:45
</textarea> </textarea>
</div><input type='hidden' id='xowa.files.cache.info__key_box' value='xowa.files.cache.info'> <input type='hidden' id='xowa.files.cache.info__ctx_box' value='app'> </div><input type='hidden' id='xowa.files.cache.info__key_box' value='xowa.files.cache.info'> <input type='hidden' id='xowa.files.cache.info__ctx_box' value='app'>
</div> </div>

2
home/wiki/Options/Firefox_addon.html
View File

@ -276,7 +276,7 @@ li.active a, li.active a:hover
</div> </div>
</div> </div>
<p> <p>
gplx.Gfo_invk__noop@7c584ba gplx.Gfo_invk__noop@70b02e04
</p> </p>
</div> </div>
</div> </div>

4
home/wiki/Options/HTML_databases.html
View File

@ -126,11 +126,11 @@
<div id='xowa.wiki.hdumps.read_preferred__name' class='xocfg_itm_name'> <div id='xowa.wiki.hdumps.read_preferred__name' class='xocfg_itm_name'>
Prefer HTML Databases for Read tab Prefer HTML Databases for Read tab
</div> </div>
<div id='xowa.wiki.hdumps.read_preferred__undo' data-xocfg-type='bool' class='xocfg_itm_delete'> <div id='xowa.wiki.hdumps.read_preferred__undo' data-xocfg-type='bool' class='xocfg_itm_delete xocfg_itm_hide'>
<span class='xoimg_btn_x16 xoimg_list_undo' onclick='return xo.cfg_edit.delete__send("xowa.wiki.hdumps.read_preferred")' title="reset to &quot;y&quot;">&nbsp;</span> <span class='xoimg_btn_x16 xoimg_list_undo' onclick='return xo.cfg_edit.delete__send("xowa.wiki.hdumps.read_preferred")' title="reset to &quot;y&quot;">&nbsp;</span>
</div> </div>
<div class='xocfg_itm_data'> <div class='xocfg_itm_data'>
<input id="xowa.wiki.hdumps.read_preferred" data-xocfg-key="xowa.wiki.hdumps.read_preferred" data-xocfg-type="bool" accesskey="d" class="xocfg_data__bool" type="checkbox"> <input id="xowa.wiki.hdumps.read_preferred" data-xocfg-key="xowa.wiki.hdumps.read_preferred" data-xocfg-type="bool" accesskey="d" class="xocfg_data__bool" type="checkbox" checked="checked">
</div><input type='hidden' id='xowa.wiki.hdumps.read_preferred__key_box' value='xowa.wiki.hdumps.read_preferred'> <input type='hidden' id='xowa.wiki.hdumps.read_preferred__ctx_box' value='app'> </div><input type='hidden' id='xowa.wiki.hdumps.read_preferred__key_box' value='xowa.wiki.hdumps.read_preferred'> <input type='hidden' id='xowa.wiki.hdumps.read_preferred__ctx_box' value='app'>
</div> </div>
<div id='xowa.wiki.hdumps.read_preferred_help_div' class='xohelp_div'> <div id='xowa.wiki.hdumps.read_preferred_help_div' class='xohelp_div'>

2
home/wiki/Options/XOWA_viewer.html
View File

@ -276,7 +276,7 @@ li.active a, li.active a:hover
</div> </div>
</div> </div>
<p> <p>
gplx.Gfo_invk__noop@7c584ba gplx.Gfo_invk__noop@70b02e04
</p> </p>
</div> </div>
</div> </div>