Introducing Chinese Text Analyser

[imron]

It's a known (to me) bug when there is a single line document without a newline at the end.

It's a rare enough use case that I haven't fixed it yet.

Thanks for reporting it though.

[realmayo]

Okay. It was just odd that I could only reproduce the error with 夜猫!

[imron]

Try this:  Select several words on a single line.  Copy and paste in to a new CTA document.  Hover the mouse over the last word in the line.

 

You'll see the same thing.  The mouse hover/click currently miscalculates the last thing in the document.  This is most often an 'end of line' character so it's not a problem.  Occasionally there are documents that don't end with an end-of-line marker, and in those cases CTA won't highlight/enable clicks for the last word of those documents.

[Yadang]

 

The mouse hover/click currently miscalculates the last thing in the document.

 

Interesting. Not sure if this is at all related, but I've noticed a couple of times that sometimes it seems like CTA thinks the words are higher/lower than they actually are. So instead of being able to select with the tip of the mouse pointer, it's more like the middle of the mouse pointer. I think in those instances I just had to close CTA and start it up again and everything was fine (maybe even just open a new document without closing CTA - can't remember). Anyways, it doesn't happen often at all, but I thought I'd mention it...

[imron]

If it happens again, send me a copy of the document that caused it and if possible, reproduction steps.

[realmayo]

in those cases CTA won't highlight/enable clicks for the last word of those documents.

 

Okay, got that.

 

But the thing with 夜猫 isn't just the inability to highlight/enable clicks: it's the stats on the right

 

 

Paste 野猫 (not the magic 夜猫!)

 

- as you say, can't click the word

- word statistics: Total: 1; known 1; % known: 100%; unknown 0

 

Paste 战猫 (not a word but two characters)

 

- as you say, can't click the last character

- word statistics: Total: 2; known 2; % known: 100%; unknown 0

 

Paste 夜猫

 

- as you say, can't click the word

- word statistics: Total: 0; known 0; % known: 100%; unknown 0

 

Edit: obviously it's no big deal! I was partly curious though because while reproducing the error, CTA froze on my twice and had to be restarted, and I couldn't remember the programme doing that to me before.

[imron]

word statistics: Total: 0; known 0; % known: 100%; unknown 0

That is a problem.  From a correctness point of view, missing words are not good.

 

CTA freezing on you is also not good.  If you can provide more specific reproduction steps that would be useful.

[Naphta]

Would it be possible to for me to replace the HSK information with TOCFL? I would prefer to have words getting their TOCFL level specified, which I use as a reference. And maybe TOCFL statics as well?

[imron]

Definitely possible.  Is there a good reference for TOCFL levels somewhere?

[Naphta]

Here's a link to the official, updated TOCFL list: http://www.sc-top.org.tw/download/8000zhuyin.zip

The excel file contains 7966 words divided into five levels. The last level (流利級) corresponds to TOCFL 5 and 6.

[imron]

Thanks.  Should be able to include that in the next release.

[Naphta]

So great! Thanks a lot ^_^

[imron]

A new version is now live!

 

There are a number of big features in this that I hope will have made it worth the wait.

 

First up is native Linux support.  Ubuntu 14 is the minimum supported version, but it should run on any x86-based machine that has Qt >= 5.2 and glibc >= 2.19.  Let me know if it doesn't and I can probably sort it out without much trouble.

 

Next is Lua scripting support.  This feature is actually really exciting, because it opens up the ability for people to write custom scripts to manipulate words and statistics from Chinese text.  This means people will be able to extend CTA without needing to wait for me to get around to adding a particular feature. I've started another thread specifically to discuss Lua scripting in CTA here.  The documentation (currently only available in the Help menu from CTA, and not online) has more details about the functionality exposed.  @realmayo - one of the example scripts provided does what you asked above about marking words as 'unknown' in normal text (it surrounds them with _underscores_).

 

The other big feature is support for multiple vocabulary profiles.  You can now have multiple different profiles and quickly and easily switch between them.  This means you can have one profile for your main vocabulary, and then a bunch of other profiles for anything else you find interesting.

 

There are a bunch of other smaller features too:

 

* Documentation (at last -available from the Help menu in the program and online).

* Copy and paste from the statistics view

* Online lookup of words

* Customisable font size for the word list view (need to edit the ChineseTextAnalyser/data/config file though)

* various new shortcut keys (see the release notes for more info).

* statistics and wordlists for TOCFL

* a bunch of minor bug fixes.

 

All in all, it's a pretty big release with a bunch of new features.  I hope you find them useful.

[艾墨本]

I tried downloading it twice but the Mac version gets a "image not recognized" warning. I don't know how to get passed this.

[murrayjames]

Imron,

 

Awesome update! When I received an email announcing your new post, I was actually reading a novel in CTA! After 2+ years, I still use the program almost every day

 

The online dictionaries are a great new feature. I'm surprised, though, that you didn't include 百度词典 as a default dictionary. For me (in Canada), Baidu is much faster than ZDIC, MDBG, and LINE. And for users in China, Baidu would be much, much faster!

 

For anyone interested, the Baidu dictionary path is:

[baidu]

   link = http://dict.baidu.com/s?wd=%s

   order = 50

 

 

Incidentally, you can't use Chinese characters in the dictionary name. I am not sure if that is intended behavior. I tried putting 百度词典 as the dictionary name and it displayed as gobbledygook in CTA.

 

Thank you Imron for including documentation with this release. I read through it. Really well done. I did notice a tiny formatting issue. On the Online Lookup web page...

 

file:///C:/Program%20Files/ChineseTextAnalyser/docs/online-lookup.html

 

...the file paths don't look right. They appear without slashes or spaces:

C:Users<username>AppDataLocalChineseTextAnalyserdataonline-search
C:Program FilesChineseTextAnalyserdataonline-search

 

Since I'm posting, I may as well make a couple feature requests

 

My #1 coveted feature is the ability to add dictionary definitions in the program itself. Currently this is a real pain. Let's say I have added 50 custom words in a month, and I want to add dictionary definitions for them. Here are the steps:

 

1. Open words.u8
2. Look up the 50 custom words in a dictionary
3. Copy or write the definitions one by one into a text editor
4. Double-check that each entry is in CEDICT format
5. Open cedict_ts.u8
6. Copy and paste my new entries

This is more work than it sounds, especially with lots of entries. It would be much more convenient to input definitions on the fly—i.e., fill out a pop-up form in CTA—whenever I add a custom word.

 

Here's a minor point. The issue might be with me   but I'll share it because it has tripped me up a few times.

 

Currently there are two different messages when closing an unsaved text. The first appears when you click [X] to close an open tab with unsaved text in it. The second appears when you close CTA and a tab with unsaved text in it is currently open. These are the messages:

 

1. The contents of this document were copied from the clipboard. Would you like to save to a file before closing? [Default: Yes]
2. You currently have unsaved documents open. Are you sure you wish to exit? [Default: Yes]

This catches my brain every time. So much so that I must think twice whenever I close a tab or the program.  I think it is because the two default "Yes" options do opposite things. One saves and closes; the other closes without saving. I am probably also used to Microsoft Word behavior, where attempting to close the program with unsaved work will ask if you want to save it, rather than confirm you want to exit without saving.

 

One final small thing. I love that CTA automatically bookmarks your location in a document. But it only does this with texts that are already saved. If you copy a text into CTA, read for a while, then close and save it, CTA will not remember your location.

 

This can be a pain if, say, you paste a novel into CTA and immediately start reading it. When you're done, you click [X] and save your document. CTA will not remember your place in this instance.

 

Imron, thank you for continuing to improve CTA! I hope you sell a million copies  

[imron]

I tried downloading it twice but the Mac version gets a "image not recognized" warning. I don't know how to get passed this.

It sounds to me like you have a corrupted download - twice, what are the chances right?

 

But I just downloaded it again now for Mac on Sierra and it worked fine.  One thing you can do to check is this.

 

First open up Terminal.

 

Then assuming you saved the file in ~/Downloads, type 

md5 ~/Downloads/chinese-text-analyser-0.99.16-install.dmg

The output should be something like:

 

MD5 (/Users/<your username>/Downloads/chinese-text-analyser-0.99.16-install.dmg) = 3ac6444404c314af83850dd9ad3eda73

 

The important part is the the number at the end.  It should be 3ac6444404c314af83850dd9ad3eda73

 

If the number is not the same, then the download has failed somehow.

@murrayjames, I'll write a longer response in a bit but just wanted to say

 

I hope you sell a million copies

Me too! Sales will have to pick up quite a bit for that to happen though

[imron]

I'm surprised, though, that you didn't include 百度词典 as a default dictionary

 

It slipped my mind.  Those others were the first three I could think of.  I'll put in in the next release.

 

 

Incidentally, you can't use Chinese characters in the dictionary name.

 

You can, you just need to make sure you save the file as UTF-8.  See attached pic.

 

...the file paths don't look right. They appear without slashes or spaces:

Good catch.  This will be fixed in the next release.

 

My #1 coveted feature is the ability to add dictionary definitions in the program itself.

The squeaky wheel gets the grease.  This has been on my todo list for ages.  I'll bump it up a few notches.

 

This catches my brain every time. So much so that I must think twice whenever I close a tab or the program.  I think it is because the two default "Yes" options do opposite things.

Excellent point.  I'll look to change this in the next version.

 

But it only does this with texts that are already saved. If you copy a text into CTA, read for a while, then close and save it, CTA will not remember your location.

Sounds like a bug.  It should be an easy fix.

 

Thanks for the excellent feedback.  Be sure to let me know if there's anything else you think could improve things.

[JamesLam]

a wild guess..do you change your IE page into Big-5under the view--->Encoding--->Chinese traditional Big-5 before you type your chinese?

[murrayjames]

Imron,

 

Thanks for your quick reply!

 

One thing I forgot to mention: Thank you for adding the dictionary lookup shortcut (Shift + double-click). With that and the toggle words known/unknown shortcut (double-click), I'm finding it even easier to move through a document. Who knew that right-clicking and selecting from a drop-down menu could take up so much time? 

 

 

a wild guess..do you change your IE page into Big-5under the view--->Encoding--->Chinese traditional Big-5 before you type your chinese? 

 

JamesLam, I'm not sure if that question is for me. I fixed it by following Imron's advice above: re-saving the "online-search" file in UTF-8 format. (A weird aside: I had to do this in Notepad. I tried saving in UTF-8 format in Notepad++, but for some reason it did not work.) 

[imron]

Documentation for all platforms is now online.