Subj : Re: Need volonteers to test another patch
To   : Nicholas Boel
From : Vitaliy Aksyonov
Date : Sat Mar 02 2024 10:42 pm

Hello Nicholas.

02 Mar 24 14:44, you wrote to me:

 VA>> Most probably it was some combination which made it looking
 VA>> almost correct. I think that screen may be the reason you saw
 VA>> pseudo-graphics

 NB> I'm no longer using screen or tmux, and this version (along with all
 NB> previous versions since you made the ncurses change) work the same and
 NB> show UTF-8 properly, except for what you describe below:

 VA>> more or less correctly. Remember those line wraps? That happens
 VA>> because GoldEd converts those symbols to UTF-8 first. All
 VA>> pseudo-graphics symbols represented as 3 bytes. So that line
 VA>> become 3 times longer in bytes. Then GoldEd tries to split
 VA>> message to lines and it uses bytes! not symbols. That's why it
 VA>> splits the line in the middle of those pseudo-graphics. Even
 VA>> worse, it may tear apart one UTF-8 symbol to two lines and it
 VA>> will be displayed incorrectly.

 NB> Yeah, I've noticed most of this.. and thank you for your explanation.
 NB> At least now I know why it is happening.

 VA>> GoldEd cannot work correctly with multibyte sequences. And even
 VA>> if it looks "correct", it's just because most English letters has
 VA>> same codes in cp437 and UTF-8.

 NB> Maybe simple ones, like german umlauts and whatnot. But cp437 doesn't
 NB> have any Cyrillic, Greek, Japanese, Chinese, etc.

 VA>> If you want to keep using UTF-8, I may only suggest to find
 VA>> version, which "works" for you and stick to it.

 NB> I already have!

 VA>> Until full UTF-8 support implemented in GoldEd (if that ever
 VA>> happen), don't expect it to work correctly, sorry.

 NB> That's ok. I had it somewhat working for awhile, the latest reverts
 NB> have changed that. I don't have an issue going back to a "lucky"
 NB> version. ;)

 VA>> It's your choice. Just be aware, that if it works - it's just
 VA>> pure luck and don't expect it to last. Until we implement UTF-8
 VA>> support. It may take years. Or never happen. It's not so easy to
 VA>> do it with backward compatibility wih all older systems like DOS
 VA>> or OS/2.

 NB> I don't mind being lucky sometimes.

I played little bit more with the code and different settings and found what 
was the difference between those two versions.

First - you use "wide" ncurses - ncursesw. I use ncurses.
Second. GoldEd incorrectly initializes ncurses in reverted version (like it was 
before I started to make any changes in GoldEd code). ncurses documentation 
explicitly says that before call initscr(), locale has to be set with 
setlocale. Otherwise it will use incorrect settings. Most probably plain C 
locale. I kinda get picture very similar to yours.

Did you have an issue with non-English chars when scrolling the message? What I 
see in UFT-8 terminal now - most of unicode text displayed correctly and 
pseudo-graphic too. Only pseudo-graphics lines broken similar to yours. And 
when I scroll text down - it corrupts.

If you may try to add one line of code to latest master and try it - that would 
be helpful.

goldlib/gcui/gkbdbase.cpp
line 149, right before initscr add this:

setlocale(LC_ALL, "");

If that gives you expected result - I'll push this change to master.

Hope you still want to dig this. :)

Vitaliy

--- GoldED+/LNX 1.1.5-b20240223
 * Origin: Aurora, Colorado (1:104/117)

.