Blog
Contents
- 1 Release: v3.4.2.1 (2016-04-10 21:00 Sun)
- 2 Release: v3.4.1.1 (2016-04-04 20:00 Sun)
- 3 Release: NONE (2016-04-03 22:00 Sun)
- 4 Release: v3.3.4.1 (2016-03-27 19:30 Sun)
- 5 Release: NONE (2016-03-20 22:00 Sun)
- 6 Release: v3.3.2.1 (Andriod only) (2016-03-13 22:00 Sun)
- 7 Release: v3.3.1.1 (Andriod only) (2016-03-06 22:00 Sun)
- 8 Release: NONE (2016-02-28 20:30 Sun)
- 9 Release: NONE (2016-02-21 19:15 Sun)
- 10 Release: NONE (2016-02-14 19:30 Sun)
- 11 Release: v3.2.1.1 (2016-02-06 20:45 Sun)
-
12 Release: v3.1.5.1 (2016-01-31 22:00 Sun)
- 12.1 Luaj fixes for references in 2016-01-13 English Wikipedia
- 12.2 Fix for German Wikipedia and extremely wide pie charts
- 12.3 New PageBanner extension for Wikivoyage
- 12.4 (Android) Show navboxes and other tables
- 12.5 (Android) Always center table content
- 12.6 (Android) Show hieroglyphics
- 12.7 (Android) Show images with ? in file title
- 12.8 Next release: v3.2.1
- 13 Release: v3.1.4.2 (2016-01-28 11:00 Thu)
-
14 Release: v3.1.4.1 (2016-01-24 19:30 Sun)
- 14.1 Scribunto / Luaj fixes for English Wiktionary
- 14.2 Better HTML dump generation for English Wiktionary, Wikisource, Wikiquote, etc. Also, French Wikipedia
- 14.3 (Android) More responsive search
- 14.4 (Android) Instant image loading when reloading page
- 14.5 (Android) Fix for pages failing to load if they have spaces and colons in the title
- 14.6 Next release: v3.1.5
- 15 Release: v3.1.3.1 (2016-01-17 21:45 Sun)
- 16 Release: XOWA (None); XOWA Android (v0.1.2.0) (2016-01-10 20:30 Sun)
- 17 Release: v3.1.1.1 (2016-01-03 20:00 Sun)
- 18 Archives
Release: v3.4.2.1 (2016-04-10 21:00 Sun)
The desktop app is a minor release. It has a fix for slow search database builds and some better Wikia support
The Android app is a major release. It has new pages for importing a wiki. It is the release candidate for XOWA beta on Google Play later this week.
(Desktop) Faster search database build
As per https://github.com/gnosygnu/xowa/issues/53 William was unable to build the new search databases on his machine. I tried it on an older dev machine, and found that it could take upwards of 5 hours to complete.
I optimized some of the SQL code and reduced it to about 2 hours. That said, note the following:
- Fails on my 32-bit Windows XP dev machine: I don't know why. The same build works on 2 Windows 7 64-bit machines, 1 openSUSE Linux 64-bit machine, and 1 Mac OS X 64-bit machine. It also works on the 32-bit JVM on the Windows 7 64-bit machine. However, the lone 32-bit Windows machine fails, and fails in a very strange way (it fails silently in release mode, and hangs in debug mode). I'm going to hope this is the non-dominant behavior, and reinvestigate at a later point in time.
- Really benefits from an SSD: SQLite really does better with an SSD. I tried some tricks like changing PRAGMA synchronous to OFF, but this only shaves off a dozen or so minutes. In contrast, an SSD will easily knock off an hour if not more (from 2 hours to about 40 min)
- If you want the new search, downloading from archive.org is the easiest way: There are other complexities with the search database rebuild (PageRank algorithm and the 10 GB pagelinks dump: See App/Search/Overview). As such, the easiest way is to download prebuilt copies from archive.org:
(Desktop) New English Wikipedia images update (2016-03)
I uploaded new consolidated versions of English Wikipedia images (2016-03) to archive.org. See https://archive.org/details/Xowa_enwiki_latest.
In terms of usage, pick one of the following:
- New user: Download the new 2016-03 version. Previously you would have to download 2015-06 set, and then download separate updates for every month between 2015-07 and 2016-03. This was just too cumbersome.
- Existing user (2015-06): Ignore the 2016-03 version. You downloaded the 2015-06 set and presumably have been keeping up with the monthly updates. The 2016-03 set is the same as your set, except it is smaller (it no longer has deleted images). If you want to remove about 8 GB of deleted images, see Dashboard/Files/Deletion. Once you remove the deleted images, you will have a set that is exactly the same as the 2016-03 version.
- Existing user (2013-11 or something else): Download the 2016-03 version. You have an old set of XOWA image databases that I really can't support any longer (they have an extension of .sqlite3). I know downloading 80 GB is painful, but I'm not planning to change database formats again in the future.
Finally, after I release XOWA Beta for Android, I'm going to work on a new update feature that will more gracefully handle monthly updates. In the meantime, please bear with the manual process.
(Desktop) Minor Wikia parser tags
A user requested some parser functionality for Wikia wikis (see https://en.wikipedia.org/wiki/User_talk:Gnosygnu . This release now supports the <choose> and <tabber> extensions.
(Android) New pages for "Import new wiki" and "XOWA Wikis"
The last major piece for XOWA Beta was more usable screens for importing wikis and managing wikis. Both these pages are now part of the latest release:
- XOWA Wikis: The "XOWA" button in the left-nav now opens a page that lists all wikis. It also allows you to import a new one by pressing the "+" icon.
- Import new wikis: The "Import" screen got a new design overhaul to make it much prettier and more intuitive. A lot of thanks goes to anonymous4 for helping with layout and feedback.
(Android) Minor fixes for crashes / obsoleted items
There were two minor crashes that would occur when using the left-nav button. The main one was pressing "Random" when on the "Import" screen
I also removed two features: "Nearby" and "Log in to Wikipedia". Neither of these seem useful for offline XOWA, and I didn't want them taking up screen real-estate.
Next release: v3.4.3
I'm planning on releasing XOWA Android Beta on Google Play mid-week. I'm sure that there will be spillover work that leads to another Android app update next week.
For wikis, the Wikimedia dump server is stalled again. I'm going to wait for it to resume before continuing with German Wikipedia.
Release: v3.4.1.1 (2016-04-04 20:00 Sun)
The desktop app is a minor release. It has a few fixes related to old .sqlite3 wikis and offline imports.
The Android app has not been changed.
(Desktop) Fixes for old .sqlite3 wikis
As reported in https://sourceforge.net/p/xowa/discussion/defects/thread/f142ca23/ v3.3.4 did not work for old .sqlite3 wikis. These wikis were phased out last year (in 2015-04), but I didn't want to drop support for them entirely.
v3.4.1 has 2 fixes for the following:
- Open old .sqlite3 wikis: Previously, XOWA would just show a blank page
- Allow old .sqlite3 wikis to upgrade to the new search system: Previously, XOWA would keep prompting to upgrade
(Desktop) Fix for offline import of large wikis
This was reported in a separate ticket: https://github.com/gnosygnu/xowa/issues/51 Basically, the new search system caused offline imports to fail for large wikis. A large wiki is defined as one with a dump file of 1.5 GB.
v3.4.1 fixes this issue. I tested with English Wikisource 2016-03: http://dumps.wikimedia.org/enwikisource/20160305/
(Desktop) Command to shrink English Wikipedia images by 8 GB
The total size of English Wikipedia is about 90 GB. 8 GB is from old images: images removed from articles during the day-to-day editing churn.
If you have English Wikipedia and have been following along with the updates, go to Dashboard/Files/Deletion and follow the instructions therein.
(Package) Release of French Wikipedia 2016-03
I had a request to update French Wikipedia, but have been delaying it for several weeks. I updated French Wiki and uploaded it here: https://archive.org/details/Xowa_frwiki_latest
Next release: v3.4.2
I'm planning to work on Android-only matters and release a "beta" app on Google play. For wikis, I'll update German Wikipedia.
Release: NONE (2016-04-03 22:00 Sun)
Sory all. I'm postponing the release for tomorrow night. I'll post again tomorrow evening. Thanks.
Release: v3.3.4.1 (2016-03-27 19:30 Sun)
The desktop app is a major release. It has a new autocomplete address bar, a new search system as well as minor parser fixes.
The Android app is a major release. It has more updates for the new search system.
(Desktop) New autocomplete address bar
The address bar now supports autocompletion. For example, open up English Wikipedia, and start typing earth
. The autocomplete is very fast and supports a lot of advanced functionality. For example you can do a search like earth + (history, future) -middle
directly in the address bar. See App/Search/Overview as well as Options/Search suggest.
Unfortunately, the autocomplete uses the new search system, so old wikis need to upgrade the search databases. You can upgrade by any of the following:
-
Use the search suggest box
- Open any page in the wiki
- Click the search suggest box (the box between "View HTML" and the magnifying glass)
- Start typing. XOWA will prompt you to upgrade the wiki.
-
Use the Dashboard/Wiki_maintenance page
- Go to Dashboard/Wiki_maintenance
- Find your wiki in the list
-
Click the
Search
button in the same row -
Click
Run Script
- Download a copy (only for users with 2016-03 English Wikipedia)
(Desktop) New search system
I completely rewrote the search system over the past several weeks. Most of these changes were meant for Android to handle fast searches with a slower processor. This new engine is now part of the desktop app, so desktop users get the same benefits as well. To summarize, here are the main benefits:
-
Faster searches: Searches are now dramatically faster for single-words (
earth
), wildcards (e*
), multiple words (earth history
) and multiple wildcards (earth* history*
) - PageRank score: Searches now use PageRank to order results. These provide better quality results than using page length. The old system would show "List of" pages at the top of the page
-
Punctuation friendly: Searches can be done for
U.S.A.
orUSA
;dont't
ordont
;half-life
orhalf
orlife
. -
Boolean searches: Searches can use boolean operators like
-
,,
,+
,()
, and*
. In addition, there's even an escape operator:\
. See App/Search/Overview -
Home wiki: The home wiki is now searchable. Try entering in
search
or any other word
(Desktop) Minor fixes for Category pages and taxonomy url links
There are basically two fixes:
- Category pages had script error: 2016-03 English Wikipedia Category pages would have script errors in the top right corner of the page. For example, https://en.wikipedia.org/wiki/Category:Articles_lacking_sources_from_February_2009
- Taxonomy boxes would have non-clickable links: 2016-03 English Wikipedia pages on biological species have a taxonomy box at the bottom of the page. It has external links which should be clickable, but were not. For example, https://en.wikipedia.org/wiki/Golden_eagle
(Android) Improvements for search system
The Android search system received more performance and accuracy improvements. Also, the Android version supports boolean searches as well. If you're using the new search system, you should definitely update your copy:
- Download the new apk: https://github.com/gnosygnu/xowa/releases
- Download the new wiki databases: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-03-05_html_core.7z
If you haven't tried XOWA on Android, please give it a try. See https://github.com/gnosygnu/xowa/wiki/Android.
Next release: v3.4.1
I've spent a lot of time on the search system for the past several weeks. I'm going to take a break and work on the following items for next week's release:
- Shrink the image databases for English Wikipedia : English Wikipedia is getting close to 128 GB but there are about 25 GB of unused image files. I'm planning on adding a feature that allows users to delete these old images. I'll also upload a brand new set.
- Resume image dumps: I'm going to upload a new French Wikipedia.
- Make minor Android changes: There are still some minor changes I need to make for the upcoming Google Play release in April.
Release: NONE (2016-03-20 22:00 Sun)
I'm postponing this week's release. I'm almost done with the desktop integration of the new search system, but I still want to make a few more changes and feel that a few more days would help.
I did upload new updates for English Wikipedia this week. See https://archive.org/download/Xowa_enwiki_latest and the 2016-03-05 files. I'll update the links later to point to them.
I'll try to upload a new Android release later this week. This will mostly have small tweaks for the new search system.
Release: v3.3.2.1 (Andriod only) (2016-03-13 22:00 Sun)
Again, there is no new release for the desktop app. I will make one next week, and resume wiki generation then.
The Android app is a major release. The major change is an improved search system. To get it, do the following:
-
Download the new Android apk: https://github.com/gnosygnu/xowa/releases/tag/v3.3.2.1
- Note: I changed the application ID yet again, so make sure you remove the old XOWA app, or you will have 2 XOWA apps. If you do end up with 2 of them, remove both, and reinstall v3.3.2.1 again
- Download the new core English Wikipedia files (1.2 GB): https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-02-04_html_core_v3.7z
Here is a list of the changes:
- "Wikipedia Alpha" updates no longer replaces XOWA: XOWA is a fork of the Wikipedia app. I haven't changed all the Wikipedia branding and IDs. In this particular case, I had left the package name the same. Whenever "Wikipedia Alpha" pushed out a new update, Android would prompt the user to install it. Installing it would remove XOWA and replace it with the "Wikipedia Alpha".
- Better performance for multi-word searches: I had to add more columns and indexes to support faster lookup for certain types of searches. For example, "earth a". Unfortunately this required updating the core English Wikipedia files again
- Do not scroll to top of results when adding new results: XOWA works by finding 2 - 5 results at a time and sending them incrementally to the search window. Each send would cause the search results to scroll back to the top. This was annoying when browsing through the list while search is running.
- Return 10 results by default and scroll to retrieve more results: Previously, XOWA would only return 50 results. Now, XOWA is smart enough to search for more as needed. Also, performance is improved by returning 10 instead of 50.
- Return search text as first result: XOWA now uses "page rank" to sort the results. However, most times, the search term may not be the highest ranking. For example, searching for "ear" would list "earth" over "ear" because "earth" has a higher rank. Now, XOWA will list "ear" first since that is the current search text
- Identify redirect articles with "→": Wikipedia uses redirects to handle items like "page renames and "misspellings". For example, "Www" redirects to "World Wide Web". XOWA better identifies these entries by showing "Www → World Wide Web"
- Remove redundant redirect articles from results: Ordinarily, there can be many redirects leading to the same page. For example, "World wide web" and "World-wide web" are both redirects to "World Wide Web". Previously, searching for "world wide web" would return all three pages. Now only 1 is returned.
- Search for entire word by putting space at end of search: XOWA automatically adds a wildcard to every search. For example, searching for "earth" actually searches for "earth*" which will return "Earth" and "Earthquake". However, sometimes only "Earth" is wanted, not "Earthquake". Adding a space and searching for "earth " will automatically limit the search to "Earth"
- Highlight search terms in search results: XOWA was previously relying on Wikipedia code to highlight search terms. This approach was rather simple: it would highlight a term only if the exact search string was found in the text. For example, "Earth History" would correctly highlight "Museum of Earth History" but not "History of Earth". This version now highlights the second as "History of Earth"
Again, I strongly recommend you get this version. There are a few bugs to work out, but this version is much improved over previous ones. This is the last major update for Android search, and future changes will probably not have as much of a noticeable impact -- at least until I get around to full-text search.
Release: v3.3.1.1 (Andriod only) (2016-03-06 22:00 Sun)
There is no new release for the desktop app. I'm still making major changes to the Search system and need more time for review.
The Android app is a major release. The major change is a faster search system. To get it, do the following:
-
Download the new Android apk: https://github.com/gnosygnu/xowa/releases/tag/v3.3.1.1
- Note: I changed the application ID, so make sure you remove the old XOWA app, or you will have 2 XOWA apps. If you do end up with 2 of them, remove both, and reinstall v3.3.1.1 again
- Download the new core English Wikipedia files (1.6 GB): https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-02-04_html_core_v2.7z
Here is a brief summary of the advantages:
- Better performance: Search is much quicker. One word searches should come back subsecond and multiple words should come back in a few, with initial results coming back in the first second
- Better ranking: Search now ranks results by page rank. Previously, XOWA ranks by page length, which resulted in long "List of" articles appearing near the top.
- Better search terms: Search now handles punctuation better. Searches can be done for "U.S.A." or "USA"; "don't" or "dont"; "half-life" or "half" or "life".
I'll explain these in greater detail in the next desktop release.
If you are using the Android beta, I strongly recommend you get this version. The new search is dramatically improved and well worth the effort.
Release: NONE (2016-02-28 20:30 Sun)
Another week, another postponement. My apologies, but I'm usually very cautious when making database changes -- especially since it's pretty important to get things right the first time. I'm going to try for a Thursday release, but may fall back on next Sunday.
I did upload the new Android English Wikipedia for 2016-02 this week. The new urls are at https://github.com/gnosygnu/xowa/issues/36#issuecomment-172428070 .
Release: NONE (2016-02-21 19:15 Sun)
I'm skipping the release this week also. I've made some good progress with search, and have a prototype that is significantly faster than the existing version. However, it'll take me another week before I add it to Android and probably two weeks before the desktop.
I uploaded the image update for English Wikipedia 2016-02 this week. You can grab the latest at https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-02-03_file_v2b.7z My next task is to make these updates easier, but I'm afraid it'll have to wait until I'm done with major Android features. I'm also postponing other image dumps as I just want to focus solely on search and related matters. I'm hoping to resume next week.
I am going to upload the HTML for English Wikipedia 2016-02 this week. If you want to try the new search, you should start downloading this version. Once I'm done with search, you'll have to download these two files to have the new upgraded search for Android:
- en.wikipedia.org-core.xowa
- en.wikipedia.org-xtn.search.core.xowa
Until then, if you're using the Android beta, please bear with the slow search. Thanks.
Release: NONE (2016-02-14 19:30 Sun)
I'm skipping the release this week. I'm trying to make a lot of performance-related changes to Search that will affect the desktop and Android version. There's a chance I may skip next week also.
Also, I'm building English Wiki 2016-02 now. I'm still a little behind in wiki dumps and I'll resume this week.
Release: v3.2.1.1 (2016-02-06 20:45 Sun)
Release new image update for English Wikipedia (2016-01)
I was a little late this month due to build issues. The direct link is https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-01-13_file_v2b.7z
Release new wikis for English Wikis (2016-01)
I've also updated all English Wikis. I generated Android versions of each as well, and will upload during the week if they proof well. The links are at https://archive.org/details/Xowa_enwiki_latest and they are listed by domain (EX: https://en.wiktionary.org)
Fix for search not working
This was a bug that I introduced back in 3.1.1. Title search doesn't work. For example: https://en.wikipedia.org/wiki/Special:Search?search=Earth&fulltext=y .
If you've built a wiki with any 3.1.* version, you will have to rebuild it. You can do this with the following:
- Go to Dashboard/Wiki_maintenance
- Find your wiki and click the "Search" button
- Scroll down to the bottom of the page and click "Run script"
You can also rebuild the entire wiki.
Fix for English Wiktionary sections not expanding correctly
This bug occurs when opening up any English Wiktionary page. Each page will have Translation tables with a "Hide" / "Show" button. The following occurred when viewing these pages (for example: https://en.wiktionary.org/wiki/green)
- Table shows up collapsed, but button says "Hide"
- Clicking on "Hide" only changes the button to "Show"
- Clicking on "Show" finally expands the table
v3.2.1 now expands these tables by default. Note that you can control the default behavior at Options/Wiki_HTML under Collapsible
New Raspberry Pi build
This build was made possible by yonisolo. Many thanks to him for taking the initiative to get this to work. As a result of his efforts, you can now use your Raspberry Pi to run XOWA as an HTTP server.
Note that the GUI does not work due to SWT issues with ARM. I don't know if anyone is interested in having this work, but if so, let me know, and I'll revisit.
Android (No release)
I'm skipping an Android release this week. I added only one-change (pinch-zoom and bottom panels), and it didn't seem worth a release.
Next release: v3.2.2
I was planning to improve the Android search version this week, but ended up spending some time on the English Wiki builds as well as underlying database code.
I'm still going to work on this database code this week. My aims are the following:
- Add a feature to auto-sync the current English Wikipedia images set (they're around 90 GB)
- Add some indices to improve performance on the Android search.
For wikis, I'm going to work on French Wiki.
Release: v3.1.5.1 (2016-01-31 22:00 Sun)
The desktop app is a major release. It resolves reference errors in 2016-01 English Wikipedia, fixes wide images on German Wikipedia, and adds a new PageBanner extension
The Android app is also a major release. It shows more tables, heiroglyphics, and images with a ? in title.
Luaj fixes for references in 2016-01-13 English Wikipedia
The 2016-01-13 dump had many erros in the reference section due to bugs in Luaj. One of the following would be displayed:
- Check |archiveurl= value (help)
- Specials character in |{some word}
These were bugs in Luaj with the frontier pattern and multi-byte chars. They were both resolved in this release.
Fix for German Wikipedia and extremely wide pie charts
I'm extremely thankful for this report at https://github.com/gnosygnu/xowa/issues/38 There were HTML issues in German Wikipedia because all the wikis rely implicitly on American formatting rules for decimals. For example:
- The German Wikipedia page for Hausmannstätten specifies a pie chart with a width (Breite) of 0.9 . See: https://de.wikipedia.org/w/index.php?title=Hausmannst%C3%A4tten&action=edit
-
This value gets multiplied by various numbers in the template. For example:
width:{{#expr:18*{{{Breite}}} }}em;
. See: https://de.wikipedia.org/w/index.php?title=Vorlage:Sitzverteilung/&action=edit
XOWA was doing this multiplication using the user's locale.
-
For US users (like me),
18*.9
was calculated correctly as16.2
-
For German users though,
18*.9
was calculated incorrectly as162
.-
This was because Java was interpreting the
.9
using German number-format rules. Since "." is the thousands separator, and "," is the decimal separator,.9
became9
. For US users, imagine,9
becoming9
.
-
This was because Java was interpreting the
v3.1.5 fixes this issue by asserting that all decimal-parsing is done in the US locale.
New PageBanner extension for Wikivoyage
This was an extension introduced by Wikimedia late last year to standardize the display of the image, icons and table of contents at the top of the page. This involved some considerable work on my side (including a Mustache parser), but XOWA now reproduces the same look.
The Wikipedia Mobile app deliberately hides navboxes as well as other tables. I think this is done for the following reasons:
- These tables were meant for desktop display, and often had wide widths
- These tables clutter the page.
In contrast, I really do like showing as much content as possible, so I decided to show these tables. I'll probably put an option to disable this later, but I'm going to wait first for some feedback from others
(Android) Always center table content
This is another deviation from the Wikipedia Mobile app style. Basically most tables are left-aligned and would end before reaching the right-hand side of the page. I changed the CSS to have these tables fill up the entire page and center the content.
This has one unfortunate sideback in that it breaks the "Collapse" buttons. I'll try to fix this in a future release, but if I can't, I may remove them entirely.
(Android) Show hieroglyphics
The early versions of the XOWA Android app did not show hieroglyphics. This is now supported in this build
(Android) Show images with ? in file title
This was a bug wherein images that had a ? were not being displayed within the Android WebView. I fixed this by replacing ? with %3F.
Next release: v3.2.1
I ran into several issues with the 2016-01 English Wikipedia dump that set me back a few days. I'm planning to upload English Wikipedia images and HTML this week, and then continue on with the other English wikis
For code, I'm going to tackle performance on the Android search again as well as a few minor UI issues on Android (collapse-table, pinch-zoom)
Release: v3.1.4.2 (2016-01-28 11:00 Thu)
This is an interim release for the desktop app. It fixes script error messages in 2016-01-13 English Wikipedia
Scribunto fixes for 2016-01-13 English Wikipedia
I accidentally had the wrong dump in place when kicking off the 2016-01-13 build. As such, I didn't spot the issue till this morning after I restarted the build.
Basically, the 2016-01-13 English Wikipedia dump introduced new page admininistration functions. These include {{PROTECTIONEXPIRY}} as well as a Scribunto function called getStabilitySettings. This resulted in a bold red "Script error" message at the top of every protected page. There are a lot of protected pages.
v3.1.4.2 resolves both issues. So far, it's stable across 35,000+ pages, but if you run across any issues, please report them. Thanks.
Release: v3.1.4.1 (2016-01-24 19:30 Sun)
The desktop app is a major release. It resolves several wiktionary errors on non-English pages and has better HTML dump generation for other wikis.
The Android app is a major release. It has a more responsive search, instant image loading when reloading page, and a fix for pages failing to load if they have spaces and colons in the title.
Scribunto / Luaj fixes for English Wiktionary
There were several dozen pages with script errors in English Wiktionary. They basically fell into these categories:
- Luaj / Scribunto errors with MCBS entities in Asian languages (1 glyph; 2 char).
-
Programmatic errors for minor cases
- Recursive gsub calls with lua function
- Search position returned as int not string
The build log for English Wiktionary looks a lot cleaner now, though I still have to add some support for the {{categorytree}} function.
Better HTML dump generation for English Wiktionary, Wikisource, Wikiquote, etc. Also, French Wikipedia
I ran the HTML dump generator through all the English wikis, as well as French Wikipedia. The results looked good, but there were a lot of log errors that were addressed with this release.
(Android) More responsive search
Search is still slow on Android. I'll fix this in a future relase, but I'll have to change the format of the search databases.
In the interim, I hacked in the following approach. Assume a search term of "Earth":
- Search for titles that match the term. For example, "Earth". This is fast and usually comes back in 0 - 3 seconds.
- Search for titles that have the term in any word within the title. For example, "History of Earth". This is moderate and usually comes back in 5 - 10 seconds.
- Search for titles that have the term starting in any word within the title. For example, "Earthquake". This is slow and usually takes 15 seconds and sometimes more.
(Android) Instant image loading when reloading page
First, some background. XOWA loads pages in two stages:
- Show the HTML
- Fill in the images one by one
The 2nd stage is necessary because it takes a few tenths of a second to retrieve an image from the database. For large pages (20+ images), this would delay page load by a few seconds (for example: 20 images * .2 second = 4 seconds).
However, once a page is viewed, reloading a page shouldn't have to re-retrieve the image from the database. The previous releases were actually re-retrieving the image. This release just uses the image if it's already there.
(Android) Fix for pages failing to load if they have spaces and colons in the title
This only affects a small minority of items that have a colon and spaces in the title. For example, "List of A:B items". XOWA didn't load this page because the underlying Wikipedia Mobile App was encoding the spaces as +.
Next release: v3.1.5
I haven't uploaded wikis this week, because I'm working on a blocking item in Wikivoyage (Pagebanner). I'm going to delay another week and focus on updating 2016-01-13 English Wikipedia. This means a new HTML dump which should be available by next Sunday. I'm also hoping to upload the other English wikis, and then continue on to French, etc.
Meanwhile, I'm going to delay the Google Play Store a few weeks. If anyone thinks I should go to Google Play Store earlier, please let me know. From my side, there are a number of items that still need polish. If you have any feedback, please do post in https://github.com/gnosygnu/xowa/issues/36 I'd appreciate anything: from "Works fine" to "This feature needs some more work".
Finally, if you have an Android device and want a wiki for your device, let me know which, and I'll bump it up in priority. The next items in queue are French, German, Chinese and Arabic Wikipedia. I'll get to all the wikis but it may take a few months unless I hear requests otherwise.
Release: v3.1.3.1 (2016-01-17 21:45 Sun)
The desktop app is a trivial release. Its main feature is a help page for generating HTML-dumps
The Android app is a major release. It has a Random feature, shows more images, and adds CSS-tweaks.
Documentation for html-dump script. See Dev/Command-line/Dumps
This item is self-explanatory. The XOWA Android app is getting more stable, so I felt it would be time to document the generation of the HTML databases.
Improved Android beta, including Random, more image display and CSS-tweaks. See https://github.com/gnosygnu/xowa/issues/36
This build has a number of new features.
- Random now works in offline mode.
- All thumb and gallery images should now show up. (approximately 5% of them weren't before)
- CSS tweaks were added to center the images as well as standardize some background colors
You can get the latest release here: https://github.com/gnosygnu/xowa/releases/
Next release: v3.1.4
The Wikimedia dump service resumed, but I'd like to delay French, Polish and Latin for another week. I'm planning to generate HTML dumps for the rest of the English wikis including Wiktionary, Wikisource, Wikivoyage, etc. This will be a good stress test for the new XOWA html-dump format before I continue on to other wikis.
I'm also going to continue tweaking the XOWA Android app. If you have a moment to help out with testing, please set it up and give me some feedback: https://github.com/gnosygnu/xowa/issues/36
Release: XOWA (None); XOWA Android (v0.1.2.0) (2016-01-10 20:30 Sun)
There's no release for the XOWA app tonight. I've spent the week mostly on the Android app, and there were no real changes for the main desktop app.
I have released an Alpha of the XOWA Android App. You should check the following links:
- Android progress thread: https://github.com/gnosygnu/xowa/issues/36
- Main Github repo (source): https://github.com/gnosygnu/xowa_android
- Releases page (apk): https://github.com/gnosygnu/xowa_android/releases/tag/v0.1.2.0
- Simple Wikipedia Android dump: https://archive.org/details/Xowa_simplewiki_latest
If you have Android KitKat or higher and would like to run XOWA on it, I'd be greatly interested in feedback. See the Android progress thread for more details.
Next release: v3.1.3
I still have some tasks for XOWA Android that I'd like to work on this week. I am going to try to release the main app next week, but if there are no new changes it may slip again in favor of an update of the XOWA Android release
I haven't generated any wikis this week as the dump service seems to have come to a halt. Whenever it resumes, I'll pick up again with the promised updates of French, Polish and Greek
Release: v3.1.1.1 (2016-01-03 20:00 Sun)
v3.1.1 is a minor release. It has a few fixes for some English Wikipedia pages, and many trivial HTML changes.
Handle refs inside {{#tag}}
I only saw this issue on one page in English Wikipedia (United Kingdom), but it has potential for more effects. Basically, XOWA did not handle <ref> inside a {{#tag}}. For example: {{#tag:ref|<ref>a</ref>}}. For the United Kingdom page, this threw off the format, and made the Infobox as wide as the entire page.
Fix for incorrect search word counts and IPA links
There was a minor issue with incorrect search counts for words. I don't think this affects anything for desktop searches, but it did throw off the Android search.
Meanwhile, due to incorrect URL generation, IPA links would not download from the web after being clicked.
Trivial HTML changes
v3.1.1 has a bunch of HTML changes to make it slightly closer to MediaWiki's HTML.
- The most significant one is the removal of the magnify icon, which is now being handled in CSS as opposed to a separate <img> tag.
- There are a few other ones affecting incorrect escaping of quotes. This would cause incomplete titles when hovering over images or links.
- Finally there are minor whitespace / attribute generation issues. None of these should be noticeable; they were made to handle log messages during bulk HTML generation
Next release: v3.1.2
I was on track to release the new Android version tonight, but ran into a last minute issue with thumbs and English Wikipedia. I'm pushing this out another week.
For wikis, I'll work on the French Wikipedia refresh, and try to add on Polish and Greek.
Finally, an apology to those I've promised minor changes. I've spent a lot of time on Android recently, and there's been a backlog of items. I'm hoping to be done with the Android version in the next few weeks and will work on these items as I get time.