Blog: Items and events from XOWA's development

From XOWA: a free, open-source, offline wiki application

Release: v3.3.4.1 (2016-03-27 19:30 Sun)

The desktop app is a major release. It has a new autocomplete address bar, a new search system as well as minor parser fixes.

The Android app is a major release. It has more updates for the new search system.

(Desktop) New autocomplete address bar

The address bar now supports autocompletion. For example, open up English Wikipedia, and start typing earth. The autocomplete is very fast and supports a lot of advanced functionality. For example you can do a search like earth + (history, future) -middle directly in the address bar. See [[App/Search/Overview]] as well as [[Options/Search suggest]].

Unfortunately, the autocomplete uses the new search system, so old wikis need to upgrade the search databases. You can upgrade by any of the following:

(Desktop) New search system

I completely rewrote the search system over the past several weeks. Most of these changes were meant for Android to handle fast searches with a slower processor. This new engine is now part of the desktop app, so desktop users get the same benefits as well. To summarize, here are the main benefits:

  • Faster searches: Searches are now dramatically faster for single-words (earth), wildcards (e*), multiple words (earth history) and multiple wildcards (earth* history*)
  • PageRank score: Searches now use PageRank to order results. These provide better quality results than using page length. The old system would show "List of" pages at the top of the page
  • Punctuation friendly: Searches can be done for U.S.A. or USA; dont't or dont; half-life or half or life.
  • Boolean searches: Searches can use boolean operators like -, ,, +, (), and *. In addition, there's even an escape operator: \. See [[App/Search/Overview]]
  • Home wiki: The home wiki is now searchable. Try entering in search or any other word

(Desktop) Minor fixes for Category pages and taxonomy url links

There are basically two fixes:

(Android) Improvements for search system

The Android search system received more performance and accuracy improvements. Also, the Android version supports boolean searches as well. If you're using the new search system, you should definitely update your copy:

If you haven't tried XOWA on Android, please give it a try. See https://github.com/gnosygnu/xowa/wiki/Android.

Next release: v3.4.1

I've spent a lot of time on the search system for the past several weeks. I'm going to take a break and work on the following items for next week's release:

  • Shrink the image databases for English Wikipedia : English Wikipedia is getting close to 128 GB but there are about 25 GB of unused image files. I'm planning on adding a feature that allows users to delete these old images. I'll also upload a brand new set.
  • Resume image dumps: I'm going to upload a new French Wikipedia.
  • Make minor Android changes: There are still some minor changes I need to make for the upcoming Google Play release in April.

Release: NONE (2016-03-20 22:00 Sun)

I'm postponing this week's release. I'm almost done with the desktop integration of the new search system, but I still want to make a few more changes and feel that a few more days would help.

I did upload new updates for English Wikipedia this week. See https://archive.org/download/Xowa_enwiki_latest and the 2016-03-05 files. I'll update the links later to point to them.

I'll try to upload a new Android release later this week. This will mostly have small tweaks for the new search system.

Release: v3.3.2.1 (Andriod only) (2016-03-13 22:00 Sun)

Again, there is no new release for the desktop app. I will make one next week, and resume wiki generation then.

The Android app is a major release. The major change is an improved search system. To get it, do the following:

Here is a list of the changes:

  • "Wikipedia Alpha" updates no longer replaces XOWA: XOWA is a fork of the Wikipedia app. I haven't changed all the Wikipedia branding and IDs. In this particular case, I had left the package name the same. Whenever "Wikipedia Alpha" pushed out a new update, Android would prompt the user to install it. Installing it would remove XOWA and replace it with the "Wikipedia Alpha".
  • Better performance for multi-word searches: I had to add more columns and indexes to support faster lookup for certain types of searches. For example, "earth a". Unfortunately this required updating the core English Wikipedia files again
  • Do not scroll to top of results when adding new results: XOWA works by finding 2 - 5 results at a time and sending them incrementally to the search window. Each send would cause the search results to scroll back to the top. This was annoying when browsing through the list while search is running.
  • Return 10 results by default and scroll to retrieve more results: Previously, XOWA would only return 50 results. Now, XOWA is smart enough to search for more as needed. Also, performance is improved by returning 10 instead of 50.
  • Return search text as first result: XOWA now uses "page rank" to sort the results. However, most times, the search term may not be the highest ranking. For example, searching for "ear" would list "earth" over "ear" because "earth" has a higher rank. Now, XOWA will list "ear" first since that is the current search text
  • Identify redirect articles with "→": Wikipedia uses redirects to handle items like "page renames and "misspellings". For example, "Www" redirects to "World Wide Web". XOWA better identifies these entries by showing "Www → World Wide Web"
  • Remove redundant redirect articles from results: Ordinarily, there can be many redirects leading to the same page. For example, "World wide web" and "World-wide web" are both redirects to "World Wide Web". Previously, searching for "world wide web" would return all three pages. Now only 1 is returned.
  • Search for entire word by putting space at end of search: XOWA automatically adds a wildcard to every search. For example, searching for "earth" actually searches for "earth*" which will return "Earth" and "Earthquake". However, sometimes only "Earth" is wanted, not "Earthquake". Adding a space and searching for "earth " will automatically limit the search to "Earth"
  • Highlight search terms in search results: XOWA was previously relying on Wikipedia code to highlight search terms. This approach was rather simple: it would highlight a term only if the exact search string was found in the text. For example, "Earth History" would correctly highlight "Museum of Earth History" but not "History of Earth". This version now highlights the second as "History of Earth"

Again, I strongly recommend you get this version. There are a few bugs to work out, but this version is much improved over previous ones. This is the last major update for Android search, and future changes will probably not have as much of a noticeable impact -- at least until I get around to full-text search.

Release: v3.3.1.1 (Andriod only) (2016-03-06 22:00 Sun)

There is no new release for the desktop app. I'm still making major changes to the Search system and need more time for review.

The Android app is a major release. The major change is a faster search system. To get it, do the following:

Here is a brief summary of the advantages:

  • Better performance: Search is much quicker. One word searches should come back subsecond and multiple words should come back in a few, with initial results coming back in the first second
  • Better ranking: Search now ranks results by page rank. Previously, XOWA ranks by page length, which resulted in long "List of" articles appearing near the top.
  • Better search terms: Search now handles punctuation better. Searches can be done for "U.S.A." or "USA"; "dont't" or "dont"; "half-life" or "half" or "life".

I'll explain these in greater detail in the next desktop release.

If you are using the Android beta, I strongly recommend you get this version. The new search is dramatically improved and well worth the effort.

Release: NONE (2016-02-28 20:30 Sun)

Another week, another postponement. My apologies, but I'm usually very cautious when making database changes -- especially since it's pretty important to get things right the first time. I'm going to try for a Thursday release, but may fall back on next Sunday.

I did upload the new Android English Wikipedia for 2016-02 this week. The new urls are at https://github.com/gnosygnu/xowa/issues/36#issuecomment-172428070 .

Release: NONE (2016-02-21 19:15 Sun)

I'm skipping the release this week also. I've made some good progress with search, and have a prototype that is significantly faster than the existing version. However, it'll take me another week before I add it to Android and probably two weeks before the desktop.

I uploaded the image update for English Wikipedia 2016-02 this week. You can grab the latest at https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-02-03_file_v2b.7z My next task is to make these updates easier, but I'm afraid it'll have to wait until I'm done with major Android features. I'm also postponing other image dumps as I just want to focus solely on search and related matters. I'm hoping to resume next week.

I am going to upload the HTML for English Wikipedia 2016-02 this week. If you want to try the new search, you should start downloading this version. Once I'm done with search, you'll have to download these two files to have the new upgraded search for Android:

  • en.wikipedia.org-core.xowa
  • en.wikipedia.org-xtn.search.core.xowa

Until then, if you're using the Android beta, please bear with the slow search. Thanks.

Release: NONE (2016-02-14 19:30 Sun)

I'm skipping the release this week. I'm trying to make a lot of performance-related changes to Search that will affect the desktop and Android version. There's a chance I may skip next week also.

Also, I'm building English Wiki 2016-02 now. I'm still a little behind in wiki dumps and I'll resume this week.

Release: v3.2.1.1 (2016-02-06 20:45 Sun)

Release new image update for English Wikipedia (2016-01)

I was a little late this month due to build issues. The direct link is https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-01-13_file_v2b.7z

Release new wikis for English Wikis (2016-01)

I've also updated all English Wikis. I generated Android versions of each as well, and will upload during the week if they proof well. The links are at https://archive.org/details/Xowa_enwiki_latest and they are listed by domain (EX: https://en.wiktionary.org)

Fix for search not working

This was a bug that I introduced back in 3.1.1. Title search doesn't work. For example: https://en.wikipedia.org/wiki/Special:Search?search=Earth&fulltext=y .

If you've built a wiki with any 3.1.* version, you will have to rebuild it. You can do this with the following:

  • Go to [[Help:Wiki maintenance]]
  • Find your wiki and click the "Search" button
  • Scroll down to the bottom of the page and click "Run script"

You can also rebuild the entire wiki.

Fix for English Wiktionary sections not expanding correctly

This bug occurs when opening up any English Wiktionary page. Each page will have Translation tables with a "Hide" / "Show" button. The following occurred when viewing these pages (for example: https://en.wiktionary.org/wiki/green)

  • Table shows up collapsed, but button says "Hide"
  • Clicking on "Hide" only changes the button to "Show"
  • Clicking on "Show" finally expands the table

v3.2.1 now expands these tables by default. Note that you can control the default behavior at [[Help:Options/Wiki HTML]] under Collapsible

New Raspberry Pi build

This build was made possible by yonisolo. Many thanks to him for taking the initiative to get this to work. As a result of his efforts, you can now use your Raspberry Pi to run XOWA as an HTTP server.

Note that the GUI does not work due to SWT issues with ARM. I don't know if anyone is interested in having this work, but if so, let me know, and I'll revisit.

Android (No release)

I'm skipping an Android release this week. I added only one-change (pinch-zoom and bottom panels), and it didn't seem worth a release.

Next release: v3.2.2

I was planning to improve the Android search version this week, but ended up spending some time on the English Wiki builds as well as underlying database code.

I'm still going to work on this database code this week. My aims are the following:

  • Add a feature to auto-sync the current English Wikipedia images set (they're around 90 GB)
  • Add some indices to improve performance on the Android search.

For wikis, I'm going to work on French Wiki.

Release: v3.1.5.1 (2016-01-31 22:00 Sun)

The desktop app is a major release. It resolves reference errors in 2016-01 English Wikipedia, fixes wide images on German Wikipedia, and adds a new PageBanner extension

The Android app is also a major release. It shows more tables, heiroglyphics, and images with a ? in title.

Luaj fixes for references in 2016-01-13 English Wikipedia

The 2016-01-13 dump had many erros in the reference section due to bugs in Luaj. One of the following would be displayed:

  • Check |archiveurl= value (help)
  • Specials character in |{some word}

These were bugs in Luaj with the frontier pattern and multi-byte chars. They were both resolved in this release.

Fix for German Wikipedia and extremely wide pie charts

I'm extremely thankful for this report at https://github.com/gnosygnu/xowa/issues/38 There were HTML issues in German Wikipedia because all the wikis rely implicitly on American formatting rules for decimals. For example:

XOWA was doing this multiplication using the user's locale.

  • For US users (like me), 18*.9 was calculated correctly as 16.2
  • For German users though, 18*.9 was calculated incorrectly as 162.
    • This was because Java was interpreting the .9 using German number-format rules. Since "." is the thousands separator, and "," is the decimal separator, .9 became 9. For US users, imagine ,9 becoming 9.

v3.1.5 fixes this issue by asserting that all decimal-parsing is done in the US locale.

New PageBanner extension for Wikivoyage

This was an extension introduced by Wikimedia late last year to standardize the display of the image, icons and table of contents at the top of the page. This involved some considerable work on my side (including a Mustache parser), but XOWA now reproduces the same look.

(Android) Show navboxes and other tables

The Wikipedia Mobile app deliberately hides navboxes as well as other tables. I think this is done for the following reasons:

  • These tables were meant for desktop display, and often had wide widths
  • These tables clutter the page.

In contrast, I really do like showing as much content as possible, so I decided to show these tables. I'll probably put an option to disable this later, but I'm going to wait first for some feedback from others

(Android) Always center table content

This is another deviation from the Wikipedia Mobile app style. Basically most tables are left-aligned and would end before reaching the right-hand side of the page. I changed the CSS to have these tables fill up the entire page and center the content.

This has one unfortunate sideback in that it breaks the "Collapse" buttons. I'll try to fix this in a future release, but if I can't, I may remove them entirely.

(Android) Show hieroglyphics

The early versions of the XOWA Android app did not show hieroglyphics. This is now supported in this build

(Android) Show images with ? in file title

This was a bug wherein images that had a ? were not being displayed within the Android WebView. I fixed this by replacing ? with %3F.

Next release: v3.2.1

I ran into several issues with the 2016-01 English Wikipedia dump that set me back a few days. I'm planning to upload English Wikipedia images and HTML this week, and then continue on with the other English wikis

For code, I'm going to tackle performance on the Android search again as well as a few minor UI issues on Android (collapse-table, pinch-zoom)

Release: v3.1.4.2 (2016-01-28 11:00 Thu)

This is an interim release for the desktop app. It fixes script error messages in 2016-01-13 English Wikipedia

Scribunto fixes for 2016-01-13 English Wikipedia

I accidentally had the wrong dump in place when kicking off the 2016-01-13 build. As such, I didn't spot the issue till this morning after I restarted the build.

Basically, the 2016-01-13 English Wikipedia dump introduced new page admininistration functions. These include {{PROTECTIONEXPIRY}} as well as a Scribunto function called getStabilitySettings. This resulted in a bold red "Script error" message at the top of every protected page. There are a lot of protected pages.

v3.1.4.2 resolves both issues. So far, it's stable across 35,000+ pages, but if you run across any issues, please report them. Thanks.

Release: v3.1.4.1 (2016-01-24 19:30 Sun)

The desktop app is a major release. It resolves several wiktionary errors on non-English pages and has better HTML dump generation for other wikis.

The Android app is a major release. It has a more responsive search, instant image loading when reloading page, and a fix for pages failing to load if they have spaces and colons in the title.

Scribunto / Luaj fixes for English Wiktionary

There were several dozen pages with script errors in English Wiktionary. They basically fell into these categories:

  • Luaj / Scribunto errors with MCBS entities in Asian languages (1 glyph; 2 char).
  • Programmatic errors for minor cases
    • Recursive gsub calls with lua function
    • Search position returned as int not string

The build log for English Wiktionary looks a lot cleaner now, though I still have to add some support for the {{categorytree}} function.

Better HTML dump generation for English Wiktionary, Wikisource, Wikiquote, etc. Also, French Wikipedia

I ran the HTML dump generator through all the English wikis, as well as French Wikipedia. The results looked good, but there were a lot of log errors that were addressed with this release.

(Android) More responsive search

Search is still slow on Android. I'll fix this in a future relase, but I'll have to change the format of the search databases.

In the interim, I hacked in the following approach. Assume a search term of "Earth":

  • Search for titles that match the term. For example, "Earth". This is fast and usually comes back in 0 - 3 seconds.
  • Search for titles that have the term in any word within the title. For example, "History of Earth". This is moderate and usually comes back in 5 - 10 seconds.
  • Search for titles that have the term starting in any word within the title. For example, "Earthquake". This is slow and usually takes 15 seconds and sometimes more.

(Android) Instant image loading when reloading page

First, some background. XOWA loads pages in two stages:

  • Show the HTML
  • Fill in the images one by one

The 2nd stage is necessary because it takes a few tenths of a second to retrieve an image from the database. For large pages (20+ images), this would delay page load by a few seconds (for example: 20 images * .2 second = 4 seconds).

However, once a page is viewed, reloading a page shouldn't have to re-retrieve the image from the database. The previous releases were actually re-retrieving the image. This release just uses the image if it's already there.

(Android) Fix for pages failing to load if they have spaces and colons in the title

This only affects a small minority of items that have a colon and spaces in the title. For example, "List of A:B items". XOWA didn't load this page because the underlying Wikipedia Mobile App was encoding the spaces as +.

Next release: v3.1.5

I haven't uploaded wikis this week, because I'm working on a blocking item in Wikivoyage (Pagebanner). I'm going to delay another week and focus on updating 2016-01-13 English Wikipedia. This means a new HTML dump which should be available by next Sunday. I'm also hoping to upload the other English wikis, and then continue on to French, etc.

Meanwhile, I'm going to delay the Google Play Store a few weeks. If anyone thinks I should go to Google Play Store earlier, please let me know. From my side, there are a number of items that still need polish. If you have any feedback, please do post in https://github.com/gnosygnu/xowa/issues/36 I'd appreciate anything: from "Works fine" to "This feature needs some more work".

Finally, if you have an Android device and want a wiki for your device, let me know which, and I'll bump it up in priority. The next items in queue are French, German, Chinese and Arabic Wikipedia. I'll get to all the wikis but it may take a few months unless I hear requests otherwise.

Release: v3.1.3.1 (2016-01-17 21:45 Sun)

The desktop app is a trivial release. Its main feature is a help page for generating HTML-dumps

The Android app is a major release. It has a Random feature, shows more images, and adds CSS-tweaks.

Documentation for html-dump script. See [[Help:Import/Command-line/Dumps]]

This item is self-explanatory. The XOWA Android app is getting more stable, so I felt it would be time to document the generation of the HTML databases.

Improved Android beta, including Random, more image display and CSS-tweaks. See https://github.com/gnosygnu/xowa/issues/36

This build has a number of new features.

  • Random now works in offline mode.
  • All thumb and gallery images should now show up. (approximately 5% of them weren't before)
  • CSS tweaks were added to center the images as well as standardize some background colors

You can get the latest release here: https://github.com/gnosygnu/xowa/releases/

Next release: v3.1.4

The Wikimedia dump service resumed, but I'd like to delay French, Polish and Latin for another week. I'm planning to generate HTML dumps for the rest of the English wikis including Wiktionary, Wikisource, Wikivoyage, etc. This will be a good stress test for the new XOWA html-dump format before I continue on to other wikis.

I'm also going to continue tweaking the XOWA Android app. If you have a moment to help out with testing, please set it up and give me some feedback: https://github.com/gnosygnu/xowa/issues/36

Release: XOWA (None); XOWA Android (v0.1.2.0) (2016-01-10 20:30 Sun)

There's no release for the XOWA app tonight. I've spent the week mostly on the Android app, and there were no real changes for the main desktop app.

I have released an Alpha of the XOWA Android App. You should check the following links:

If you have Android KitKat or higher and would like to run XOWA on it, I'd be greatly interested in feedback. See the Android progress thread for more details.

Next release: v3.1.3

I still have some tasks for XOWA Android that I'd like to work on this week. I am going to try to release the main app next week, but if there are no new changes it may slip again in favor of an update of the XOWA Android release

I haven't generated any wikis this week as the dump service seems to have come to a halt. Whenever it resumes, I'll pick up again with the promised updates of French, Polish and Greek

Release: v3.1.1.1 (2016-01-03 20:00 Sun)

v3.1.1 is a minor release. It has a few fixes for some English Wikipedia pages, and many trivial HTML changes.

Handle refs inside {{#tag}}

I only saw this issue on one page in English Wikipedia (United Kingdom), but it has potential for more effects. Basically, XOWA did not handle <ref> inside a {{#tag}}. For example: {{#tag:ref|<ref>a</ref>}}. For the United Kingdom page, this threw off the format, and made the Infobox as wide as the entire page.

Fix for incorrect search word counts and IPA links

There was a minor issue with incorrect search counts for words. I don't think this affects anything for desktop searches, but it did throw off the Android search.

Meanwhile, due to incorrect URL generation, IPA links would not download from the web after being clicked.

Trivial HTML changes

v3.1.1 has a bunch of HTML changes to make it slightly closer to MediaWiki's HTML.

  • The most significant one is the removal of the magnify icon, which is now being handled in CSS as opposed to a separate <img> tag.
  • There are a few other ones affecting incorrect escaping of quotes. This would cause incomplete titles when hovering over images or links.
  • Finally there are minor whitespace / attribute generation issues. None of these should be noticeable; they were made to handle log messages during bulk HTML generation

Next release: v3.1.2

I was on track to release the new Android version tonight, but ran into a last minute issue with thumbs and English Wikipedia. I'm pushing this out another week.

For wikis, I'll work on the French Wikipedia refresh, and try to add on Polish and Greek.

Finally, an apology to those I've promised minor changes. I've spent a lot of time on Android recently, and there's been a backlog of items. I'm hoping to be done with the Android version in the next few weeks and will work on these items as I get time.

Release: NONE (2015-12-27 22:00 Sun)

I'm skipping the release again. There were two minor changes, but neither seem to be enough to warrant another version. As always, if anyone notices any serious issues, please do let me know at either https://github.com/gnosygnu/xowa/issues or gnosygnu at gmail.com .

I did release Russian wikis this week. See: https://archive.org/details/Xowa_ruwiki_latest

I've spent a lot of time this week on the new Android version. I've actually reached a point this weekend wherein it is working with English Wikipedia.

For 3.1.1, I'm going to try to release this new Alpha with for Simple Wikipedia. My aim is to get a release of English Wikipedia shortly afterwards.

For wikis, I'll try to do a refresh of French Wikipedia.

Release: NONE (2015-12-20 22:00 Sun)

I'm skipping the release this week as well. I've spent the week working on the new Android dump format for gallery, as well as a new database merge feature. I haven't found anything worth a release.

I posted the English image updates at https://archive.org/details/Xowa_enwiki_latest I'm working on Russian now, and they'll be ready next week.

For 2.12.4, I'm still working on the new Android version for Simple Wikipedia.

Release: NONE (2015-12-13 19:30 Sun)

I'm skipping the release this week. I've been working on Android-related changes, so there really aren't any significant changes.

I posted Arabic wikis at https://archive.org/details/Xowa_arwiki_latest I ran into an issue with English Wikipedia though, and am rebuilding the dump. It should be ready by the middle of the week.

For 2.12.3, I'm continuing work on the Android version for Simple Wikipedia. For wikis, I'll try to do Russian

Release: NONE (2015-12-13 19:30 Sun)

I'm skipping the release this week. I've been working on Android-related changes, so there really aren't any significant changes.

I posted Arabic wikis at https://archive.org/details/Xowa_arwiki_latest I ran into an issue with English Wikipedia though, and am rebuilding the dump. It should be ready by the middle of the week.

For 2.12.3, I'm continuing work on the Android version for Simple Wikipedia. For wikis, I'll try to do Russian

Release: v2.12.1.1 (2015-12-06 21:30 Sun)

v2.12.1 is a minor release. It has several fixes directly affecting Italian Wikisource pages, and a few minor parser fixes for other wikis

Wiki bundles for Italian wikis

The Italian wikis have been posted to archive.org

Minor fixes for Italian Wikisource

These fixes addressed the following:

  • Poems with references would show duplicate references at the bottom of the page. In addition, the text of the references may be incorrectly placed in the body of the poem.
  • Image maps with scribunto code relying on the page title generated script errors.

Minor parser fixes including mapsource extension

These fixes are several, but the following are notable:

  • Scribunto code calling json-decode was failing if the code was relying on integer keys. For example, {"1"="a"} needs to parse 1 as an integer, not a string. Otherwise, code calling ipairs would later fail
  • Mapsource functions such as {{#deg2dd}} would not handle non-ASCII double apostrophes. For example, ’’ needs to go to '' and then to ". XOWA was doing the 1st part ('') but not the second (")

Next release: v2.12.2

I spent a good deal of time working on most of these minor fixes, so my progress on Android was limited. However, I did get a text version of Simple Wikipedia working with the new dump and the new UI. I'm planning to work on images this week, and hope to have a new beta for v2.12.2

For wikis, I'm generating English now. I'll try to add Arabic this week as well.

Release: NONE (2015-11-29 19:30 Sun)

This week will be another skipped release. I've only made very minor changes this week, so it didn't seem worth it to make a new release.

I did post wikis for Dutch and Latin at https://archive.org/details/Xowa_nlwiki_latest and https://archive.org/details/Xowa_lawiki_latest

For 2.12.1 I'm going to finish up the dump and try to get another Android version for Simple Wikipedia. For wikis, I'll work on Italian.

Release: v2.11.4.1 (2015-11-22 20:00 Sun)

v2.11.4 is a minor release. It has one fix affecting French Wikipedia pages, and another for the simple-mode Category system.

Fix for "long-vertical scrolling" on French Wikipedia

In v2.11.2, I added a fix to handle extraneous pre sections: see #Fix for extraneous pre-sections in References. It turned out this fix was not correct, and caused pages in French Wikipedia to have large scrolling sections.

I reverted the fix for v2.11.4. As it turned out, the issue with extraneous pre sections was handled by the underlying templates themselves.

Fix for simple-mode Category pages not loading

XOWA generates a simple Category system when it builds a wiki (this is called v1). If you want an authentic version of the Category system (this is called v2), you can always import it separately under [[Import offline]].

The simple Category system looks like it broke some time ago during the general database redesign from .sqlite3 to .xowa. I fixed it for this release.

Wiki bundles for French, Lithuanian, Limburgish, Estonian, Latvian, Nynorsk, and Cebuano wikis

There were a lot of wikis this week, but they were mostly small. These have been uploaded to archive.org:

Next release: v2.11.5

I was able to do a basic HTML dump for simple wiki, but there are still a number of issues. These should be fixed this week, and hopefully I can try the new HTML dump system on English Wikipedia.

For wikis, I'm going to work through the last of the old .sqlite3 wikis and go through Dutch (nl.*) and Latin wikis.

Release: v2.11.3.1 (2015-11-15 20:00 Sun)

v2.11.3 is a fairly minor release. It addresses one minor bug with "1. REDIRECT" showing up on pages.

Fix for "1. REDIRECT" showing up on some pages

This was a bug I introduced while fixing another issue in v2.11.2. Some pages would now show "1. REDIRECT", most notably en.w:Earth

Wiki bundles for Slovak, Slovenian, Galician, and Waray-Waray wikis

These have been uploaded to archive.org:

Next release: v2.11.4

I fixed the one minor Wikibase issue, but am deferring a few other possible issues till later. I'm still chugging away on the HTML dump. My aim is to get a simple wiki dump for the next release.

I generated the English Wiki dump but there were only about 3 thousand new images, and I didn't feel that it merited an update. For wikis, I'm going to update French, and then go through as many as I can of Lithuanian, Limburgish, Estonian, Latvian and Nynorsk.

Release: v2.11.2.1 (2015-11-08 22:00 Sun)

v2.11.2 is a significant release. It addresses minor, but noticeable issues with gallery sections, location maps, and references. It also has some Wikibase fixes.

Fix for Gallery rendering vertically instead of horizontally

Wikimedia added a change a few weeks ago to force <gallery> tags to use a css. See https://phabricator.wikimedia.org/T110709 This resulted in galleries being shown vertically, instead of horizontally.

v2.11.2 now includes this css.

Fix for Location maps not working on non-English wikis

This has been a bug for a few months, but I had misdiagnosed it as bad Module code. Basically, non-English wikis would copy-paste the same Module:Location_map code in to their wikis. This failed to work in XOWA, because:

  • The Module code has a hard-coded reference to Template
  • XOWA actually generates the non-English name for Template for the wiki. For Croatian Wikipedia, this is Šablon not Template

This basically broke location maps in non-English wikis. I should have caught this earlier, but the first few I looked at were all bad usages (no location coordinates passed).

v2.11.2 now uses the canonical name just like MediaWiki.

Fix for extraneous pre-sections in References

I think this is also a recent bug, but I can't track down the actual cause. MediaWiki has inconsistent logic for trimming whitespace around template and link arguments. The one at issue was a piece of code like {{TemplateName| B\n }}. On my local version of MediaWiki (1.25.2), it doesn't trim the "\n " after the B and creates a <pre> section. However, on Wikipedia, it does trim the whitespace and does not create a <pre> section.

I think this behavior changed recently, but couldn't find the actual line of code in MediaWiki. For now, XOWA will automatically trim end whitespace for all template arguments. In my opinion this is more intuitive behavior anyway.

Minor Wikibase fixes

These are a handful of signficant fixes that only affect some several dozen pages. They are in brief:

  • in Lua code, entity.formatPropertyValues('P123') would not work. Note that entity.formatPropertyValues('property_name') did work
  • Lua code expects a "datatype" property on entity and a value of "globe-coordinate".
  • Lua code required actual numbers for quantity ("99"), not the literal Wikidata value ("+99")

Minor fixes for Redirect

These are two minor items worth more explanation:

  • Redirects would not work if they had an argument. For example, #REDIRECT [[Target_page|link=collapsed]]
  • Clicking on "Redirected from" links at the top of the page would not work. For example, try https://en.wikipedia.org/wiki/UK and click on "Redirected from UK"

Minor JTidy fix to always enclose body text

This was actually not working from the start. Basically tidy needs to take something like ... 

<div class="floatleft">a</div>a1<div class="floatleft">b</div>b1

... and add some <p> tags to produce ... 

<div class="floatleft">a</div><p>a1</p><div class="floatleft">b</div><p>b1</p>

This has a noticeable visual effect for some wikis (Croatian Wiktionary) which have HTML like the above.

I modified the jtidy_xowa.jar to handle this behavior.

Wiki bundles for Danish, Serbo-Croatian, Croatian, and Estonian wikis

These have been uploaded to archive.org

Next release: v2.11.3

I made some good progress with the rewritten HTML dump feature this week, but am still not done. I'm hoping to get ready for v2.11.3 but I need to work on a few other minor issues that have spilled over from this week. (namely in Wikibase).

For wikis, I'm generating 2015-11 English Wikipedia. I'm also going to continue through the minor wikis and try to do Slovak, Waray-Waray, Slovanian and Galician

Release: v2.11.1.1 (2015-11-01 19:30 Sun)

v2.11.1 is a minor release. There are a few minor fixes for Hebrew wikis.

Minor fixes for Hebrew wikis

v2.11.1 now supports Hebrew dates for the time function like {{#time:xhxjj xjx xhxjY|28-08-1608 + 341 days}} generating ד' ה'שס\"ט". In addition there is basic support for Hebrew grammar like {{grammar:prefixed|וabc}} generating ווabc.

Image packages for English Wikipedia, Hebrew, Norwegian and Vietnamese wikis

The English Wikipedia update for 2015-10 is finally up. The rest of the wikis are up at archive.org

Next release: v2.11.2

I've been working on a new UI for the Android alpha. I've also spent time rewriting the HTML dump. For the next few weeks, I don't expect to do anything with the desktop app outside of bug fixing.

For wikis, I'll move on to Danish, Serbo-Croatian, Hungarian and Esperanto wikis

Release: NONE (2015-10-25 20:00 Sun)

I'm skipping the release this week. I spent most of the time this week on the HTML dump as well as Android. I'm looking at doing the regular monthly release next week but don't expect any major issues.

I'm also working on the wikis below and will post during the week.

Release: v2.10.3.1 (2015-10-18 20:15 Sun)

v2.10.3 is a minor release. There are a few minor fixes for issues related to v2.10.2.

Minor fixes

There are two worth mentioning:

  • Commons wiki would not download files
  • Mediawiki would not handle the translate function

The rest are more trivial in comparison. You can check the [[Help:Change log]] for more detail.

Image packages for Serbian, Malay and Bulgarian wikis

I'm uploading Bulgarian now, but the rest should be ready at archive.org. See: https://archive.org/details/Xowa_srwiki_latest https://archive.org/details/Xowa_mswiki_latest and https://archive.org/details/Xowa_bgwiki_latest

Next release: v2.10.4

I'm still waiting on commons / wikidata before doing an English Wikipedia update. I'll work on Hebrew this week and Norwegian / Vietnamese as well.

For code, I'm going to change tack again and table the article update function. I'm working on a different XOWA Android UI with hopes of updating the alpha in the next few weeks.

Release: v2.10.2.1 (2015-10-11 22:45 Sun)

v2.10.2 is a major release. It has a critical fix for the 2015-10 English Wikipedia dump. It also features better functionality for Chinese / Serbian language wikis. Finally, it also has some significant under-the-hood changes

Fix for broken references in 2015-10 English Wikipedia dump

XOWA was showing broken references on 2.6 millions pages for the 2015-10 English Wikipedia dump. This was due to some recent changes in Module:Citation/CS1. Specifically, the number of local variables in the citation0 function was greater than the default 200 limit for Luaj.

I recompiled a new version of luaj_xowa.jar to fix this issue.

Language support for Chinese / Serbian languages

This item basically involves the following changes:

  • Serbian languages now have language dropdowns. For example, sr.wikipedia.org now allows you to switch between Cyrillic and Latin variants
  • Chinese and Serbian wikis now fully implement the "-{}-" syntax. This includes some significant functions like recursive conversion, word-interchangeability, and adhoc rule-definition. See: https://www.mediawiki.org/wiki/Writing_systems/Syntax

I'm really hoping that the new functionality matches MediaWiki. If you're a Chinese / Serbian user, please take a moment to spot-check a few pages and let me know if there is anything amiss.

Better support for <source> and <syntaxhighlight>

MediaWiki recently changed the <syntaxhighlight> extension away from geshi to Pygments. In the process, it looks like they changed a few of the general parsing rules as well. v2.10.4 has better support for these blocks, though actual syntax coloring is still some time away.

Full support for interwiki links

v2.10.2 now incorporates online calls to Wikimedia APIs. The first use was to download a full list of interwiki aliases for each wiki. For those users in offline mode, you can manually generate the list. For more info, see [[Help:Site config]]

Support for Dansguardian import filter

XOWA supports custom creation of wikis by matching articles against words. For example, you can create an English Wikipedia that doesn't have any Pokemon articles. Or you can create an English Wikipedia that just has Pokemon articles. The system is intricate but should be quite powerful. For more info, see [[Help:Import/Dansguardian]]

More source code cleanup

There was a lot of source code cleanup. The process is still ongoing as I'm working on removing old code, as well as rewriting some knottier sections.

Next release: v2.10.3

I didn't have time to generate any wikis this week. I'm going to do English, Serbian and one of Malay, Hebrew, Bulgarian for v2.10.2

For code, I'm going to start working on an article update function (update the offline article with the latest version from online Wikipedia).


Release: NONE (2015-10-04 22:00 Sun)

I'm skipping the release this week. I ran into some issues with Chinese / Serbian translations, though I think I'm pretty close to being done. In addition, I have some other significant changes in queue which I didn't want to introduce for a "first-week-of-the-month" release.

If you want a recent stable release, use v2.9.4.1. I'll keep that release up for a month.

Otherwise, next week's v2.10.2 will have more changes. I'll also resume generating wikis from the list below.

Namespaces

XOWA

Help

Donate