<ahref="#.28Wiki.29_Publish_2016-08_English_Wikipedia_and_German_Wikipedia"><spanclass="tocnumber">1.1</span><spanclass="toctext">(Wiki) Publish 2016-08 English Wikipedia and German Wikipedia</span></a>
</li>
<liclass="toclevel-2 tocsection-3">
<ahref="#.28PC_.2F_Android.29_Better_html_database_support_for_packed_galleries.2C_video_thumbnails.2C_and_image_maps"><spanclass="tocnumber">1.2</span><spanclass="toctext">(PC / Android) Better html database support for packed galleries, video thumbnails, and image maps</span></a>
<ahref="#.28PC.29_Support_the_lsth_parser_function"><spanclass="tocnumber">1.3</span><spanclass="toctext">(PC) Support the lsth parser function</span></a>
<ahref="#.28PC.29_Add_torrent_link_.2F_MD5_info_to_the_Download_Central_Info_page"><spanclass="tocnumber">1.4</span><spanclass="toctext">(PC) Add torrent link / MD5 info to the Download Central Info page</span></a>
<ahref="#.28PC.29_Fix_random_.22SSL_peer_shut_down_incorrectly.22_when_downloading_database_dumps"><spanclass="tocnumber">5.2</span><spanclass="toctext">(PC) Fix random "SSL peer shut down incorrectly" when downloading database dumps</span></a>
</li>
<liclass="toclevel-2 tocsection-15">
<ahref="#.28PC.29_Fix_.22Script_error:Module:Effective_protection_expiry.22_for_many_English_Wikipedia_pages"><spanclass="tocnumber">5.3</span><spanclass="toctext">(PC) Fix "Script error:Module:Effective protection expiry" for many English Wikipedia pages</span></a>
<ahref="#.28PC.29_Fix_parser_issues_for_Polish_Wikipedia_pages.2C_including_location_maps_and_broken_infoboxes"><spanclass="tocnumber">5.4</span><spanclass="toctext">(PC) Fix parser issues for Polish Wikipedia pages, including location maps and broken infoboxes</span></a>
<ahref="#.28PC.29_Fix_.22invalid_title.22_when_clicking_on_wikis_in_left-hand_nav"><spanclass="tocnumber">6.2</span><spanclass="toctext">(PC) Fix "invalid title" when clicking on wikis in left-hand nav</span></a>
<ahref="#.28PC.29_Minor_wikibase_update_for_.22renderSnaks.22"><spanclass="tocnumber">6.3</span><spanclass="toctext">(PC) Minor wikibase update for "renderSnaks"</span></a>
<ahref="#.28Android.29_Fix_random_failure_when_restarting_XOWA_while_downloading_wikis"><spanclass="tocnumber">6.4</span><spanclass="toctext">(Android) Fix random failure when restarting XOWA while downloading wikis</span></a>
The PC version is a major release. It adds better html database support for certain images, handles the lsth parser function, and makes minor changes to the Download Central Info page
</p>
<p>
The Android version is a major release. It also includes the same better support for certain images.
</p>
<h3>
<spanclass="mw-headline"id=".28Wiki.29_Publish_2016-08_English_Wikipedia_and_German_Wikipedia">(Wiki) Publish 2016-08 English Wikipedia and German Wikipedia</span>
The 2016-08 versions of both wikis are finally up. You can get them through <ahref="http://xowa.org/home/wiki/Special:XowaDownloadCentral.html"id="xolnki_12"title="Special:XowaDownloadCentral"class="xowa-visited">Download Central</a>.
<spanclass="mw-headline"id=".28PC_.2F_Android.29_Better_html_database_support_for_packed_galleries.2C_video_thumbnails.2C_and_image_maps">(PC / Android) Better html database support for packed galleries, video thumbnails, and image maps</span>
</h3>
<p>
There was a small percentage (< .1%) of images that didn't appear in the HTML databases. These include the following:
</p>
<ul>
<li>
<b>Packed galleries</b>: Certain galleries are marked as "packed". These galleries are dynamically resized to fill up as much space as possible. For example: <ahref="https://en.wikipedia.org/wiki/Mexico"rel="nofollow"class="external free">https://en.wikipedia.org/wiki/Mexico</a>
</li>
<li>
<b>Video thumbnails for .ogg files</b>: The .ogg extension is generally used for audio files, but a small number of video files use the .ogg extension. For example: <ahref="https://en.wikipedia.org/wiki/Lepidoptera"rel="nofollow"class="external free">https://en.wikipedia.org/wiki/Lepidoptera</a>
</li>
<li>
<b>Image map buttons</b>: Image maps usually have an information button that says "Click for more info". For example: <ahref="https://en.wikipedia.org/wiki/Philippines"rel="nofollow"class="external free">https://en.wikipedia.org/wiki/Philippines</a>
</li>
</ul>
<p>
The fix for the first two items required a major change to the HTML database format. The following is a brief (though technical) summary:
</p>
<ul>
<li>
Wikitext metadata is no longer embededded with the images; previously an HTML attribute was embedded with this info. For example, <code><img data-xowa-image="0|220|110|-1|-1"</code>.
</li>
<li>
All orig_image info is early-resolved at HTML-generation time; previously, orig_image info was late-resolved at HTML-view time.
</li>
<li>
All file-related HTML is now correctly generated at HTML-generation time; previously, it was "guessed" at HTML-generation time. Note that this "guess" was 99.9% correct, but failed for things like "a .ogg file which is actually a video"
</li>
</ul>
<p>
From a general user standpoint, this change should be entirely transparent. v3.8.5 will work with both the old HTML databases as well as the new ones. However, any version earlier v3.8.4 and earlier will not work with the new HTML databases.
</p>
<p>
Finally, the 2016-08 English and German Wikipedia are the first wikis to use the multi-threaded builder. This allowed me to cut down the English Wikipedia generation time from 80 hours to 12 hours (with 2 high-grade 8-core machines). This was a significant under-the-cover change, but hopefully most of the kinks have been worked out.
</p>
<h3>
<spanclass="mw-headline"id=".28PC.29_Support_the_lsth_parser_function">(PC) Support the lsth parser function</span>
</h3>
<p>
I discovered that a number of English Wikipedia pages used the {{#lsth}} function. In brief, this function includes a section from another page on the current page. For example: <ahref="https://en.wikipedia.org/wiki/10s_BC"rel="nofollow"class="external free">https://en.wikipedia.org/wiki/10s_BC</a> . I ended up rewriting the lst functions to support this behavior.
</p>
<h3>
<spanclass="mw-headline"id=".28PC.29_Add_torrent_link_.2F_MD5_info_to_the_Download_Central_Info_page">(PC) Add torrent link / MD5 info to the Download Central Info page</span>
Based on some welcome feedback from malakov1, I added a torrent link to the Download Central Info page. For example, <ahref="http://xowa.org/home/wiki/Special:XowaDownloadCentralInfo%3Ftask_id%3D96.html"id="xolnki_13"title="Special:XowaDownloadCentralInfo?task id=96">Special:XowaDownloadCentralInfo?task id=96</a>. Note that these torrent links are to the full archive.org folder, not the task. In particular, the following two types will be included:
<b>Old database dumps</b>: I keep at least one old set around. Otherwise, users who are downloading an old set will be suddenly cut off when a new set is published.
</li>
<li>
<b>Other wikis</b>: All wikis for a language are saved in a given folder. For example, <ahref="https://archive.org/download/Xowa_dewiki_latest/"rel="nofollow"class="external free">https://archive.org/download/Xowa_dewiki_latest/</a> has German Wikipedia, but also German Wiktionary, German Wikisource, etc.
</li>
</ul>
<p>
In most cases, you will need to manually exclude files from the torrent. Most torrent programs support this feature. For example, in qBittorrent do the following:
</p>
<ul>
<li>
Download the torrent: <ahref="https://archive.org/download/Xowa_dewiki_latest/Xowa_dewiki_latest_archive.torrent"rel="nofollow"class="external free">https://archive.org/download/Xowa_dewiki_latest/Xowa_dewiki_latest_archive.torrent</a>
</li>
<li>
Click on the torrent in the main pain
</li>
<li>
Click the "Content" button near the bottom of the window
</li>
<li>
Uncheck the files you're not interested in
</li>
</ul>
<p>
In addition, the Download Central Info page now includes MD5 and file size info.
<b>Wikis</b>: I'm going to keep this week light and work on the Czech wikis..
</li>
<li>
<b>Dynamic update of wiki pages</b>: A primitive version of this functionality is included in this release, but it's disabled by default. If you want to try it, I'll post more information at <ahref="https://github.com/gnosygnu/xowa/issues/72"rel="nofollow"class="external free">https://github.com/gnosygnu/xowa/issues/72</a> . Otherwise, I plan to finish it up for the next release.
I'm postponing the release till tomorrow night. I uploaded 2016-08 English and German Wikipedia, but there are a few minor tweaks I want to make before release.
I'm skipping the release this week. I'm still working on the 2016-08 English Wikipedia. I am close, but I still don't have a clean HTML dump yet. Please check again next week.
<b>Wikis</b>: I didn't finish 2016-08 English Wikipedia this week. I'm making a lot of significant changes to handle several minor issues. They primarily affect packed galleries, image maps, and certain video files (particularly .ogg ones). I should be on schedule for releasing 2016-08 English Wikipedia for next Monday's release.
</li>
</ul>
<ul>
<li>
<b>Dynamic update of wiki pages</b>: I didn't do anything on this issue this week as English Wikipedia took all my time. I'll put aside time for this week.
The PC version is a major release. It fixes random "SSL peer shut down incorrectly" when downloading, "Script Error:Module Protection expiry" on many en.w pages, and some parser issues on pl.w.
The 2016-07 versions of the Polish wikis are up. You can get them through <ahref="http://xowa.org/home/wiki/Special:XowaDownloadCentral.html"id="xolnki_14"title="Special:XowaDownloadCentral"class="xowa-visited">Download Central</a>.
<spanclass="mw-headline"id=".28PC.29_Fix_random_.22SSL_peer_shut_down_incorrectly.22_when_downloading_database_dumps">(PC) Fix random "SSL peer shut down incorrectly" when downloading database dumps</span>
</h3>
<p>
In the past week, XOWA would fail while downloading database dumps from the Wikimedia servers at <ahref="https://dumps.wikimedia.org/backup-index.html"rel="nofollow"class="external free">https://dumps.wikimedia.org/backup-index.html</a> . After some investigation, it looks like Wikimedia is currently resetting all connections at the 17 and 47 minute mark of every hour (EX: 1:17, 1:47, 2:17, 2:47, etc.) . For more info, see <ahref="https://lists.wikimedia.org/pipermail/xmldatadumps-l/2016-August/001325.html"rel="nofollow"class="external free">https://lists.wikimedia.org/pipermail/xmldatadumps-l/2016-August/001325.html</a>
</p>
<p>
As the issue is still ongoing, I'm going to point XOWA to default to <ahref="http://your.org"rel="nofollow"class="external free">http://your.org</a> instead. Note that your.org is an official backup as per <ahref="https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps"rel="nofollow"class="external free">https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps</a> .
</p>
<p>
If you want to change your.org to point elsewhere, do the following:
</p>
<ul>
<li>
Go to home/wiki/Options/Import
</li>
<li>
Change "Dump servers" to the following
</li>
</ul>
<pre>
https://dumps.wikimedia.org/,
http://dumps.wikimedia.your.org/,
http://wikipedia.c3sl.ufpr.br/,
http://ftp.fi.muni.cz/pub/wikimedia/
</pre>
<ul>
<li>
Press "Save"
</li>
</ul>
<p>
Many thanks to Kevin Day and your.org for their assistance and support with this issue.
</p>
<h3>
<spanclass="mw-headline"id=".28PC.29_Fix_.22Script_error:Module:Effective_protection_expiry.22_for_many_English_Wikipedia_pages">(PC) Fix "Script error:Module:Effective protection expiry" for many English Wikipedia pages</span>
</h3>
<p>
The 2016-07-20 version of English Wikipedia would show "Script error" on many pages. This was due to a minor typo on my side for "infinite" vs "infinity". It is fixed with of this release.
</p>
<p>
Thanks to William to reporting this issue.
</p>
<h3>
<spanclass="mw-headline"id=".28PC.29_Fix_parser_issues_for_Polish_Wikipedia_pages.2C_including_location_maps_and_broken_infoboxes">(PC) Fix parser issues for Polish Wikipedia pages, including location maps and broken infoboxes</span>
</h3>
<p>
Polish Wikipedia required two major changes:
</p>
<ul>
<li>
Location maps weren't showing due to issues with JSON parsing.
</li>
<li>
Some infoboxes were garbled because of incorrect parsing of <ref%> tags.
<b>Wikis</b>: I'm working on 2016-08 English Wikipedia now, and want to add a few fixes for this HTML dump release. Unfortunately, the new HTML generator is not ready yet, so English Wikipedia will probably take the full week.
</li>
<li>
<b>Dynamic update of wiki pages</b>: I added "wikimedia API" calls this week. I'm going to work on recursively parsing all templates for this week.requested in <ahref="https://github.com/gnosygnu/xowa/issues/72"rel="nofollow"class="external free">https://github.com/gnosygnu/xowa/issues/72</a> . This will probably take a few weeks. For next week, I'm going to work on the simpler "wikimedia API" calls
</li>
<li>
<b>Redlinks for HTTP Server</b>: This has kept slipping for a few weeks. I'm going to table this for now, and come back to it later.
</li>
<li>
<b>mwad integration with the status bar</b>: I'm also tabling this as well.
The 2016-07 versions of the Arabic wikis are up. You can get them through <ahref="http://xowa.org/home/wiki/Special:XowaDownloadCentral.html"id="xolnki_15"title="Special:XowaDownloadCentral"class="xowa-visited">Download Central</a>.
<spanclass="mw-headline"id=".28PC.29_Fix_.22invalid_title.22_when_clicking_on_wikis_in_left-hand_nav">(PC) Fix "invalid title" when clicking on wikis in left-hand nav</span>
This was kindly reported by markusmaresch in <ahref="https://github.com/gnosygnu/xowa/issues/73"rel="nofollow"class="external free">https://github.com/gnosygnu/xowa/issues/73</a> . Basically, clicking on a wiki in the left-hand sidebar would fail with "invalid title". This was due to some changes I made with the UI in a recent release.
Wikidata (www.wikidata.org), or "Wikibase", is a continually evolving project. French Wikipedia started using two new methods recently: renderSnaks and renderSnak. I added support for these calls in this release. I also cleaned up some Wikibase code.
<spanclass="mw-headline"id=".28Android.29_Fix_random_failure_when_restarting_XOWA_while_downloading_wikis">(Android) Fix random failure when restarting XOWA while downloading wikis</span>
A helpful anonymous user forwarded me a crash report for XOWA Android. Sometimes, XOWA Android would fail to restart when in the process of downloading wikis from Download Central. This was broken since the initial release of Download Central.
<b>New HTML database generator</b>: I'm still been plugging away at the HTML generator. It's closer to 99% now, and I'll start phasing it in over the next few releases.
<b>Dynamic update of wiki pages</b>: This was requested in <ahref="https://github.com/gnosygnu/xowa/issues/72"rel="nofollow"class="external free">https://github.com/gnosygnu/xowa/issues/72</a> . This will probably take a few weeks. For next week, I'm going to work on the simpler "wikimedia API" calls.
<li><ahref="http://dumps.wikimedia.org/backup-index.html"title="Get wiki datababase dumps directly from Wikimedia">Wikimedia dumps</a></li>
<li><ahref="https://archive.org/search.php?query=xowa"title="Search archive.org for XOWA files">XOWA @ archive.org</a></li>
<li><ahref="http://en.wikipedia.org"title="Visit Wikipedia (and compare to XOWA!)">English Wikipedia</a></li>
</ul>
</div>
</div>
<divclass="portal"id='xowa-portal-donate'>
<h3>Donate</h3>
<divclass="body">
<ul>
<li><ahref="https://archive.org/donate/index.php"title="Support archive.org!">archive.org</a></li><!-- listed first due to recent fire damages: http://blog.archive.org/2013/11/06/scanning-center-fire-please-help-rebuild/ -->
