2016-08
@@ -67,361 +67,123 @@
- Release: v3.9.4.2 (2016-09-26 20:00 Mon)
+ Release: v3.10.3.1 (2016-10-16 20:00 Sun)
- The PC version is a patch release. It fixes Categories not showing for pages with more than one word in the title.
+ The PC version is a major release. It adds better category support, fixes a Scribunto "Script error out of bounds", and a new custom scripting system.
+
+
+ The Android version is a minor release. It has a possible fix for some devices failing to load XOWA when permissions are revoked and then regranted.
- Show Catgories for pages with more than one word in the title
+ (Wiki) Release 2016-10 English and German Wikipedia
- As per https://github.com/gnosygnu/xowa/issues/81#issuecomment-249666435 Categories would not show for certain pages. It turned out this was a careless bug which affected pages with more than one word in the title.
-
-
- Thanks to Ope30 for catching this bug. There's also another one he caught regarding large categories. This will be fixed in a future release or by manually downloading patched databases. See https://github.com/gnosygnu/xowa/issues/81
-
-
- Release: v3.9.4.1 (2016-09-25 20:00 Sun)
-
-
- The PC version is a major release. It has more Category fixes, more page sync changes, and a few other changes.
-
-
- The Android version is a minor release. It has a fix for bad file downloads in Download Central.
-
-
- (Wiki) Release 2016-09 French, English and German Wikipedia
-
-
- I published the French wikis last week, but I'm just listing them formally in this release.
-
-
- The English and German wikis were published today. To get them, do the following:
+ The 2016-10 versions of both wikis are up. You can get them through:
-
- Go to home/wiki/Special:XowaDownloadCentral
+ Wiki_setup/English_wikis
-
- (if you have v3.9.3.1 or earlier) Click the "Download" icon to the right of the "Available" table
-
- -
- Look for 2016-09 entries in the "Available" table.
-
-
-
- Thanks to fake4d for pointing out the bad link before.
-
-
- (PC) More minor Category fixes including large categories, dynamic page list and wikidata.org
-
-
- v3.9.2.1 introduced a new Category system to support Categories in HTML dump mode. Unfortunately, a few items were left out.
-
-
- -
- Maximum of 200 pages: Some categories have more than 200 pages but XOWA would only show the first 200.
-
- -
- Dynamic Page List did not work: Wikibooks, Wikiversity and Wikinews use an extension called Dynamic Page List which relies heavily on Categories. These pages always came up blank.
-
- -
- Wikidata import failed: Importing wikidata.org through Import Online / Offline would fail
-
-
-
- All of these are fixed in v3.9.4.1. In addition, I also added a feature to disable "Previous 200" on the first page and "Next 200" on the last page.
-
-
- Finally, these all work with both the Wikitext dumps and the HTML dumps. I still have to get them to work on Android.
-
-
- (PC) Page sync shows math and video images
-
-
- v3.9.2.1 added a page sync feature, but it didn't handle Math, video and pdf images. v3.9.4.1 now handles these. There are still a few other image types to handle (helper buttons / score), but sync should now work fine for 99% of pages.
-
-
- (PC / Android) Download Central does not loop endlessly if wrong file size downloaded
-
-
- This is a bug that should hopefully not have affected anyone. While testing the downloads for English Wikipedia, I noticed that a "bad file size" error would basically cause Download Central to cycle endlessly. The only option would be to remove the entire task and resume.
-
-
- (PC) Show pages in MediaWiki namespace
-
-
- v3.9.4.1 now is able to show wikitext pages in the MediaWiki namespace. For example, https://en.wikipedia.org/wiki/MediaWiki:Edittools .
-
-
- (PC) Run GUI on Raspberry Pi
-
-
- As per https://github.com/gnosygnu/xowa/issues/85 the XOWA GUI was not able to run on the Raspberry Pi. I changed SWT to use WebKit in order to get this working. So far, I can only confirm that it works on my device, but hopefully it will work for others.
-
-
- In addition, this build has the official release candidate sqlite-jdbc jar as per https://github.com/xerial/sqlite-jdbc/issues/58
-
-
- Many thanks to nstents for helping me test the builds.
-
-
- (PC) More parser fixes including Scribunto regex at end of string and gaps in argument keys
-
-
- There were two significant fixes worth mentioning:
-
-
- -
- End-of-string regexes did not work: Due to a one-off array bug, XOWA wouldn't correctly process end-of-string regular expressions in Lua. So, something like
gsub( "e?$", "e", 1 ) returned wrong results
-
- -
- Gaps in argument keys would fail: Scribunto allows integer keys, but with gaps. For example, something like
{1="a", 4="d", 7="g"}. XOWA had logic to handle 1 gap, but failed if there were more
+ Wiki_setup/German_wikis
- Next release: v3.10.1
+ (PC) Multiple category changes, including performance, international sorting, numeric sorting, and HTML database loading
+
+
+ First, round of thanks to Ope30 who noticed a number of issues with the category system as decribed in https://github.com/gnosygnu/xowa/issues/87
+
+
+ This particular release has a lot of changes based on the bug reports:
+
+
+ -
+ Improved performance on SATA drives, especially for large categories: The category system in v3.9.4.1 worked okay on SSD drives, but did poorly on SATA drives. Large categories (2,000+) like https://en.wikipedia.org/wiki/Category:2001_albums would take 20-30 seconds to load. Now they take 4-5 seconds.
+
+ -
+ Next 200 / Previous 200 would not work on English Wikipedia for large categories: English Wikipedia recently changed their sorting collation from "uppercase" to "uca-default-u-kn". As a result, a number of entries no longer sorted correctly, and pressing "Next / Previous" would break. As an example, https://en.wikipedia.org/wiki/Category:All_stub_articles now works correctly navigating forward
+
+ -
+ HTML dumps now show wikitext HTML: Category pages sometimes have a letter index at the top of the page, or other information. For example, https://de.wikipedia.org/wiki/Category:Mann has a large table of contents listing many 2-letter combinations. This information didn't appear in the HTML dumps before. Now they do
+
+ -
+ Other minor changes: There are a small number of other changes as well which are covered more in the Change log
+
+
+
+ (PC) Fix for "Script error: index out of bounds" on 2016-10 enwiki pages
+
+
+ A number of pages in 2016-10 English Wikipedia would should "Script error: index out of bounds" in the Infobox area near the Coordinates. For example, https://en.wikipedia.org/wiki/Paris This was fixed in this release
+
+
+ (PC) Add custom scripting support through standalone javascript / lua files
+
+
+ This was a feature requested at https://github.com/gnosygnu/xowa/issues/88
+
+
+ Basically, v3.10.4.2 allows changing XOWA pages through custom Javascript. This is intended for non-Wikimedia wikis (such as Wikia) which use custom extension that are not supported in XOWA. A user can add this missing custom extension by writing an extension in Javascript.
+
+
+ More information is at App/Xtn/Scripting
+
+
+ Next release: v3.10.2
-
- Wikis: After a busy two weeks, I'm going to take a break this week. If there's a wiki you want, please feel free to request!
+ Wikis: I'm generating 2016-10 French Wikipedia now, and should have it uploaded in the next day. Afterwards, I'm going to go through the English sister wikis (Wiktionary, Wikisource, Wikivoyage) and then hopefully the German ones.
-
- Dynamic update of wiki pages: I still have a little work to do on this feature.
-
- -
- Image shrink: I tested a Download Central version of the image shrink but it felt too slow. I'm going to opt instead for the standalone page. I'll upload the file this week and post again in the blog.
+ Dynamic update of wiki pages: Between categories and scripting, I didn't get a chance to work on this feature
- Release: NONE (2016-09-18 22:00 Sun)
+ Release: NONE (2016-10-02 21:30 Sun)
- I'm skipping the release this week. I'm working on a few changes to categories, and English Wikipedia is spilling over a little.
+ There's no release this week. I'm working on a category performance enhancement before starting on 2016-10 wikis.
- I published French Wikipedia over the weekend. You can download them automatically with Download Central, or manually at https://archive.org/download/Xowa_frwiki_latest
+ I did create a deletion database for English Wikipedia. It will reduce the size of the images datases from 101 GB to 90 GB.
- English Wikipedia should go up over the next few days. German Wikipedia will follow afterwards.
+ The link is here: https://archive.org/download/Xowa_enwiki_latest/en.wikipedia.org-file-deletion-2016.09.xowa The instructions are here: Dashboard/Files/Deletion
Thanks.
-
- Release: v3.9.2.1 (2016-09-11 19:00 Sun)
-
-
- The PC version is a major release. It has a fix for broken Category pages, a new Category system, a new page sync feature, and other changes.
-
-
- The Android version is a minor release. It has a fix for dropped connections in Download Central.
-
-
- (PC) Fix for Category pages not working
-
-
- v3.8.5.1 had a bug that broke Category pages. For example, https://simple.wikipedia.org/wiki/Category:Earth . This would only retrieve one random category. v3.9.2.1 fixes the issue.
-
-
- (PC) New Category system to handle HTML dumps
-
-
- As per https://github.com/gnosygnu/xowa/issues/81 Category pages were not showing for HTML dumps. I rewrote the Category system to handle this. Here are the high-level points:
-
-
- -
- Works for future HTML dumps: I will be uploading new versions of 2016-09 English, German and French Wikipedia this week that will use the Category system. Unfortunately, existing HTML dumps won't work with them.
-
- -
- Smaller size: The new Category system makes some database changes to reduce file size. For English Wikipedia, that means a difference between 10 GB and 8 GB. 8 GB may sound like a lot for Categories, but keep in mind there are over 100 million page to category links.
-
- -
- Backward compatible: The existing Category system will work with the new code-base.
-
- -
- Does not work for text database dumps: XOWA originally started off storing files in text files instead of sqlite files. I switched over to SQLite three years ago and phased out text databases two years ago. It's possible that some users with old wikis (3 years old) may still have these text databases. If so, then the new Category system won't work.
-
-
-
- In short, the changes should be transparent to most, except HTML dump users will soon get Categories.
-
-
- There's a little more info at Category Overview for those interested.
-
-
- (PC) Basic page synchronization feature to get latest from online
-
-
- As per https://github.com/gnosygnu/xowa/issues/72 XOWA will now be able to sync pages directly from the online version. This feature is still a work-in-progress, and probably should only be done for selective pages. Certain pages won't sync correctly for a few more releases -- namely, pages with Math. See App/Xtn/Page_sync for more info.
-
-
- (PC / Android) Download Central changes for dropped connections / Mac OS X
-
-
- As per https://github.com/gnosygnu/xowa/issues/80 dropped connections in Download Central would appear to hang on Mac OS X. This release should now force Mac OS X to report dropped connections. In addition, it also will try to reconnect for a minute once a connection is dropped, before giving up.
-
-
- (PC) Various parser changes including Protectiong Banner, stack overflow, and invalid UTF-8
-
-
- This release has a bunch of parser changes. The Change log has a complete list. The most relevant items are the following:
-
-
- -
- Protection banners now appear: Previously, they didn't show due to a bug in returning Protection data. Currently, they will show whenever the pages says they are protected. Note that no page is ever protected from editing in XOWA: you can always click on the Edit button.
-
- -
- Stack overflow error: A handful of pages would crash the program with stack overflow. These will now just show with omitted content
-
- -
- Xowa/Luaj issue with foreign characters: There was a multi-threading issue with foreign characters. These are random and very low frequency. They will be fixed for future dumps.
-
-
-
- (PC) Other minor changes including read-only support, torrent-file links and docs for HTML dump
-
-
- Thanks to multiple users, this build also resolves some minor issues:
-
-
- -
- Read-only Sqlite files are slow (thanks to Ope30): Previously, a sqlite file marked as read-only would be much slower. This build now handles read-only files correctly.
-
- -
- Torrent links for non-Wikipedia wikis were wrong (thanks to malakov1): Download Central was reporting the wrong torrent links for non-Wikipedia wikis. For example, English Wiktionary was being listed as http://archive.org/Xowa_enwiktionary_latest . This build corrects the issue.
-
- -
- Wrong documentation for HTML dumps (thanks to Rüdiger Gleim): The command-line script for generating HTML dumps have changed over the past few releases. I forgot to update the documentation page at Dev/Command-line/Thumbs.
-
-
-
- Also, at risk of omission, additional thanks to markusmaresch for reporting the Mac OS X issue, and MDE186 for testing the page sync feature.
-
-
- Next release: v3.9.3
-
-
- -
- Wikis: This is going to be a busy week. I'm going to generate 2016-09 versions of English, German, and French Wikipedia with the new Category System and Random.
-
- -
- Dynamic update of wiki pages: I still have more work to do on this feature. In particular, I'd like to get Math and other images showing correctly.
-
- -
- Image shrink: I'm going to try to shrink the English Wikipedia images on a quarterly basis. I'm hoping to get this integrated directly into Download Central, so Android users can get it to work
-
-
-
-
-
-
- Wikis: Czech Wikis (2016-09) (2016-09-05 23:00 Mon)
-
-
- The Czech wikis are available in Download Central. To get them, do the following:
-
-
- -
- Go to home/wiki/Special:XowaDownloadCentral
-
- -
- Click the "Download" icon to the right of the "Available" table
-
- -
- The Czech wikis will appear at the bottom of the "Available" table.
-
-
-
- Release: NONE (2016-09-04 22:00 Sun)
-
-
- This week has no release. I'm working on a variety of items based on issues at https://github.com/gnosygnu/xowa/issues Thanks to Ope30, markusmarech, and some anonymous others for all their input!
-
-
- The Czech wikis are done and being uploaded now. They should be ready by tomorrow in Download Central (click the Download icon to get a new list of wikis)
-
-
- For wikis, I'm going to try to update 2016-09 English Wikipedia, but I want to work on adding Categories as well as a few other items.
-