Archives/gnosygnu_xowa
1
0
Fork 0
mirror of https://github.com/gnosygnu/xowa.git synced 2024-10-27 20:34:16 +00:00
Code Issues Projects Releases Wiki Activity
abf6c554c9
gnosygnu_xowa/wiki/home/page/Dev/Command-line/Wikidata.html

206 lines
10 KiB
HTML
Raw Normal View History

'v3.4.2.4'
2016-04-12 23:06:56 +00:00
<!DOCTYPE html>
<html dir="ltr">
<head>
<meta http-equiv="content-type" content="text/html;charset=UTF-8" />
<title>Dev/Command-line/Wikidata - XOWA</title>
'v3.4.2.27'
2016-04-16 00:13:58 +00:00
<link rel="shortcut icon" href="//gnosygnu.github.io/xowa/xowa_logo.png" />
<link rel="stylesheet" href="//gnosygnu.github.io/xowa/xowa_common.css" type="text/css">
'v3.4.2.6'
2016-04-15 20:20:49 +00:00
<style>
.console {font-family: monospace; color: #EEEEEE ; background-color: black ; border: medium solid black;}
.code
,.path
,.url {font-family: monospace; color: black ; background-color: #f9f9f9 ; border: medium solid #f9f9f9;}
.bold {font-weight: 900;}
</style>
<style>
.console {font-family: monospace; color: #EEEEEE ; background-color: black ; border: medium solid black;}
.code
,.path
,.url {font-family: monospace; color: black ; background-color: #f9f9f9 ; border: medium solid #f9f9f9;}
.bold {font-weight: 900;}
</style>
'v3.4.2.4'
2016-04-12 23:06:56 +00:00
</head>
<body class="mediawiki ltr sitedir-ltr ns-0 ns-subject skin-vector action-submit vector-animateLayout" spellcheck="false">
<div id="mw-page-base" class="noprint"></div>
<div id="mw-head-base" class="noprint"></div>
<div id="content" class="mw-body">
<h1 id="firstHeading" class="firstHeading"><span>Dev/Command-line/Wikidata</span></h1>
<div id="bodyContent" class="mw-body-content">
<div id="siteSub">From XOWA: the free, open-source, offline wiki application</div>
<div id="contentSub"></div>
<div id="mw-content-text" lang="en" dir="ltr" class="mw-content-ltr">
<div id="toc" class="toc">
<div id="toctitle">
<h2>
Contents
</h2>
</div>
<ul>
<li class="toclevel-1 tocsection-1">
<a href="#Import_using_the_XML_dump"><span class="tocnumber">1</span> <span class="toctext">Import using the XML dump</span></a>
</li>
<li class="toclevel-1 tocsection-2">
<a href="#Import_using_the_JSON_dump"><span class="tocnumber">2</span> <span class="toctext">Import using the JSON dump</span></a>
</li>
</ul>
</div>
<p>
XOWA can import Wikidata through the command-line
</p>
<h2>
<span class="mw-headline" id="Import_using_the_XML_dump">Import using the XML dump</span>
</h2>
<p>
XOWA can build wikidata using the XML dump at www.mediwa/wikidatawiki/. This import is basically the same as an import of any other wiki.
</p>
<p>
The script for the XML import follows.
</p>
<pre class='code'>
// build wikidata database; this only needs to be done once, whenever wikidata is updated
add ('www.wikidata.org' , 'util.cleanup') {delete_all = 'y';}
add ('www.wikidata.org' , 'util.download') {dump_type = 'pages-articles';}
add ('www.wikidata.org' , 'util.download') {dump_type = 'categorylinks';}
add ('www.wikidata.org' , 'util.download') {dump_type = 'page_props';}
add ('www.wikidata.org' , 'util.download') {dump_type = 'image';}
add ('www.wikidata.org' , 'text.init');
add ('www.wikidata.org' , 'text.page');
add ('www.wikidata.org' , 'text.cat.core');
add ('www.wikidata.org' , 'text.cat.link');
add ('www.wikidata.org' , 'text.cat.hidden');
add ('www.wikidata.org' , 'text.term');
add ('www.wikidata.org' , 'text.css');
add ('www.wikidata.org' , 'util.cleanup') {delete_tmp = 'y'; delete_by_match('*.xml|*.sql|*.bz2|*.gz');}
</pre>
<h2>
<span class="mw-headline" id="Import_using_the_JSON_dump">Import using the JSON dump</span>
</h2>
<p>
As of v2.6.3, XOWA also provides basic support for building wikidata from the JSON dump. This support was added for the following reasons:
</p>
<ul>
<li>
<b>Current delay in XML dumps</b>: The last good wikidata XML dump was 2+ months old due to problems with dump generation. See: <a href="https://phabricator.wikimedia.org/T98585" rel="nofollow" class="external free">https://phabricator.wikimedia.org/T98585</a>
</li>
<li>
<b>JSON dumps recommended</b>: Wikidata seems to prefer using the JSON dump over the XML dump. See: <a href="http://www.wikidata.org/wiki/Wikidata:Database_download" rel="nofollow" class="external free">http://www.wikidata.org/wiki/Wikidata:Database_download</a>
</li>
<li>
<b>JSON dumps are more frequent</b>: The JSON dumps have been dumping regularly on a weekly basis. In contrast the XML dumps take 3 - 4 weeks.
</li>
</ul>
<p>
Despite these reasons, there are limitations to the JSON dump.
</p>
<ul>
<li>
<b>Non-JSON pages not available</b>: The JSON dump doesn't provide other pages, such as the Main Page or MediaWiki pages. Only pages in the main and property namespaces are available. This is by design. See: <a href="https://lists.wikimedia.org/pipermail/wikidata/2015-June/006441.html" rel="nofollow" class="external free">https://lists.wikimedia.org/pipermail/wikidata/2015-June/006441.html</a>
</li>
<li>
<b>Page metadata not available</b> : Certain properties are not available, such as page_id and last_modified. XOWA provides substitutes for these values, but they will not match the Wikimedia version
</li>
</ul>
<p>
The script for the JSON import follows.
</p>
<pre class='code'>
add ('www.wikidata.org' , 'util.cleanup') {delete_all = 'y';}
// TODO: add ('www.wikidata.org' , 'util.download') {dump_type = 'wikidata-json';}
add ('www.wikidata.org' , 'wbase.json_dump');
add ('www.wikidata.org' , 'text.term');
add ('www.wikidata.org' , 'text.css');
add ('www.wikidata.org' , 'util.cleanup') {delete_tmp = 'y'; delete_by_match('*.xml|*.sql|*.bz2|*.gz|*.json');}
</pre>
</div>
</div>
</div>
<div id="mw-head" class="noprint">
<div id="left-navigation">
<div id="p-namespaces" class="vectorTabs">
<h3>Namespaces</h3>
<ul>
<li id="ca-nstab-main" class="selected"><span><a id="ca-nstab-main-href" href="index.html">Page</a></span></li>
</ul>
</div>
</div>
</div>
<div id='mw-panel' class='noprint'>
<div id='p-logo'>
'v3.4.2.27'
2016-04-16 00:13:58 +00:00
<a style="background-image: url(//gnosygnu.github.io/xowa/xowa_logo.png);" href="http://xowa.org" title="Visit the main page"></a>
'v3.4.2.4'
2016-04-12 23:06:56 +00:00
</div>
<div class="portal" id='xowa-portal-home'>
<h3>XOWA</h3>
<div class="body">
<ul>
'v3.4.2.27'
2016-04-16 00:13:58 +00:00
<li><a href="//gnosygnu.github.io/xowa/index.html" title='Visit the main page'>Main page</a></li>
<li><a href="//gnosygnu.github.io/xowa/screenshots.html" title='See screenshots of XOWA'>Screenshots</a></li>
<li><a href="//gnosygnu.github.io/xowa/wiki/home/page/Help/Download_XOWA.html" title='Download the XOWA application'>Download XOWA</a></li>
<li><a href="//gnosygnu.github.io/xowa/wiki/home/page/Dashboard/Image_databases.html" title='Download offline wikis and image databases'>Download wikis</a></li>
'v3.4.2.6'
2016-04-15 20:20:49 +00:00
</ul>
</div>
</div>
<div class="portal" id='xowa-portal-stargin'>
<h3>Getting started</h3>
<div class="body">
<ul>
'v3.4.2.27'
2016-04-16 00:13:58 +00:00
<li><a href="//gnosygnu.github.io/xowa/wiki/home/page/App/Setup/System_requirements.html" title='Get XOWA&apos;s system requirements'>Requirements</a></li>
<li><a href="//gnosygnu.github.io/xowa/wiki/home/page/App/Setup/Installation.html" title='Get instructions for installing XOWA'>Installation</a></li>
<li><a href="//gnosygnu.github.io/xowa/wiki/home/page/App/Import/Simple_Wikipedia.html" title='Learn how to set up Simple Wikipedia'>Simple Wikipedia</a></li>
<li><a href="//gnosygnu.github.io/xowa/wiki/home/page/App/Import/English_Wikipedia.html" title='Learn how to set up English Wikipedia'>English Wikipedia</a></li>
<li><a href="//gnosygnu.github.io/xowa/wiki/home/page/App/Import/Other_wikis.html" title='Learn how to set up other Wikipedias'>Other Wikipedias</a></li>
'v3.4.2.4'
2016-04-12 23:06:56 +00:00
</ul>
</div>
</div>
<div class="portal" id='xowa-portal-help'>
<h3>Help</h3>
<div class="body">
<ul>
'v3.4.2.27'
2016-04-16 00:13:58 +00:00
<li><a href="//gnosygnu.github.io/xowa/wiki/home/page/Help/About.html" title='Get more information about XOWA'>About</a></li>
<li><a href="//gnosygnu.github.io/xowa/wiki/home/page/Help/Contents.html" title='View a list of help topics'>Contents</a></li>
<li><a href="//gnosygnu.github.io/xowa/wiki/home/page/Help/Media.html" title='Read what others have written about XOWA'>Media</a></li>
<li><a href="//gnosygnu.github.io/xowa/wiki/home/page/Help/Feedback.html" title='Questions? Comments? Leave feedback for XOWA'>Feedback</a></li>
'v3.4.2.4'
2016-04-12 23:06:56 +00:00
</ul>
</div>
</div>
'v3.4.2.6'
2016-04-15 20:20:49 +00:00
<div class="portal" id='xowa-portal-blog'>
<h3>Blog</h3>
<div class="body">
<ul>
'v3.4.2.27'
2016-04-16 00:13:58 +00:00
<li><a href="//gnosygnu.github.io/xowa/wiki/home/page/Blog.html" title='Follow XOWA''s development process'>Current</a></li>
'v3.4.2.6'
2016-04-15 20:20:49 +00:00
</ul>
</div>
</div>
'v3.4.2.4'
2016-04-12 23:06:56 +00:00
<div class="portal" id='xowa-portal-links'>
<h3>Links</h3>
<div class="body">
<ul>
<li><a href="http://dumps.wikimedia.org/backup-index.html" title="Get wiki datababase dumps directly from Wikimedia">Wikimedia dumps</a></li>
<li><a href="https://archive.org/search.php?query=xowa" title="Search archive.org for XOWA files">XOWA @ archive.org</a></li>
<li><a href="http://en.wikipedia.org" title="Visit Wikipedia (and compare to XOWA!)">English Wikipedia</a></li>
</ul>
</div>
</div>
<div class="portal" id='xowa-portal-donate'>
<h3>Donate</h3>
<div class="body">
<ul>
<li><a href="https://archive.org/donate/index.php" title="Support archive.org!">archive.org</a></li><!-- listed first due to recent fire damages: http://blog.archive.org/2013/11/06/scanning-center-fire-please-help-rebuild/ -->
<li><a href="https://donate.wikimedia.org/wiki/Special:FundraiserRedirector" title="Support Wikipedia!">Wikipedia</a></li>
<!-- <li><a href="" title="Support XOWA! (but only after you've supported archive.org and Wikipedia)">XOWA</a></li> -->
</ul>
</div>
</div>
</div>
</body>
</html>
Reference in New Issue Copy Permalink