Re: Fw: [opensuse-m17n] Konsole displays some characters wrong

At Sat, 13 Jun 2015 16:56:57 +0900 (JST), Norman Diamond wrote:

OK, it could be 20 years too late to prevent ambiguous-width Romaji, katakana, and space characters from entering Unicode.

However, the Unicode consortium continues to add characters.  It isn't too late to add some necessary characters.

A half-width downarrow can be added with the same width as half-width space, Romaji, and katakana.  A full-width downarrow can be added with same same width as full-width space, Romaji, and katakana.  Ambiguous-width space, Romaji, and katakana can be added to help unify the performance of applications when the text doesn't happen to include arrows.

You can try argue with Unicode people, but IMO it's against the philosophy of Unicode. In the history or Unicode, such ambiguity isn't resolved by different characters but with different attributes. Please read articles about Unicode. There have been many discussions about CJK, especially glyphs shared by CJK but can be rendered differently. The full/half width is one of such arguments. But, as I already wrote, you should try globaladvance option at first. Takashi

Sincerely, Norman Diamond

Takashi Iwai wrote:

...

Norman Diamond wrote:

...

I think the Unicode consortium should be taught that full-width and

half-width downarrow can be distinguished just as full-width Romaji can be distinguished from half-width Romaji, without even needing CJK.  Even a full-width plain blank space is distinguished from a half-width plain blank space.

It's 20 years too late :)

...

However, anyone who edits these source files will likely have CJK fonts.  The comments are mostly in Japanese.  Strings have mostly been moved to data files, but changes to the code are generally accompanied by adding or editing strings that are also Japanese.

There is globaladvance attribute in the fontconfig and we used to set it false for CJK for some time.  The attribute itself is marked as deprecated nowadays, but this might still work.  You need to give it a try.

Takashi

...

Yours sincerely, Norman Diamond

Fuminobu TAKEYAMA replied to me:

...

...

  I do not agree with your statement that downarrow should be displayed as half-width.  It is still   the same character that it used to be in ShiftJIS and EUC encodings, and the adoption of Unicode   and UTF-8 does not change the width of the glyph.

The ambiguous width problem have been discussed for a long time. There is no way to distinguish CJK down arrow and non-CJK down arrow.

...

  If I the character becomes half-width when I edit source files in Konsole, but it's still   full-width when other people edit the same source files in a Gnome terminal or even in Windows,   it will result in people scrambling source files.

If they edit GNOME terminal without CJK fonts, arrows are half-width. Even if we resolve this Konsole bug, your problem will not be resolved.

You still need to keep in mind that down arrow (and some other characters) have ambiguous width.

...

  I do not know who decided that downarrow should have an ambiguous width.   If the Unicode consortium decided that, they're causing problems.

They did: http://www.unicode.org/reports/tr11/tr11-28.html#Ambiguous http://ja.wikipedia.org/wiki/東アジアの文字幅

Anyway, let's fix this Konsole's bug.

It is quite difficult to propose a new Unicode specification that can be acceptable for both CJK and non-CJK people.

Fuminobu TAKEYAMA

...

  Since the font selection dialog's preview uses Qt widgets instead of whatever Konsole uses, we   know that the full-width downarrow character is displayed as full-width by

...

  Furthermore in Suse 11.0 Konsole itself also displayed the full-width downarrow character as   full-width.  So I think this bug is in the newer version of Konsole itself, not in the rest of   Linux.  I can't say if the preview window should be removed because it doesn't really provide a   preview, but it helped narrow down this bug.

  I do not agree with your statement that downarrow should be displayed as half-width.  It is still   the same character that it used to be in ShiftJIS and EUC encodings, and

...

  and UTF-8 does not change the width of the glyph.

  If I the character becomes half-width when I edit source files in Konsole, but it's still   full-width when other people edit the same source files in a Gnome terminal or even in Windows,   it will result in people scrambling source files.

  Full-width ａ and half-width a do not have ambiguous widths.  Full-width ア and half-width ｱ do not   have ambiguous widths.  I do not know who decided that downarrow should have an ambiguous width.   If the Unicode consortium decided that, they're causing

On 2015/06/13 10:35, Norman Diamond wrote: these Linux tools. the adoption of Unicode problems.

...

  Yours sincerely, Norman Diamond

  ----- Forwarded Message -----

...

  From: Fuminobu TAKEYAMA To:

...

...

  01:30 Subject: Re: [opensuse-m17n] Konsole displays some characters wrong

>>  I still think there's a bug.

  Yes, this is clearly a Japanese specific bug. Could you file a bug report for openSUSE and   KDE?

  I saw a patch to fix the same problem with emacs about 5 years ago. So I think this problem can   be resolved by some hacks.

>>  Konsole knows that the glyph for 字 is full-width, so Konsole should know that the glyph for >>  downarrow is full-width.

  Down arrow is known as ambiguous-width character. Monospaced font should provide half-width   down arrow but some Japanese fonts such as IPA Gothic and MS Gothic do full-width down arrow   from a kind of historical reason.

>>  The version of Konsole that was included in Suse 11.0 did know

...

...

>>  full-width.  I never had to look at what the default font was because it just worked.

  Konsole's character layouting was changed recently as far as I know so that some   (flexible-width) bold fonts can be placed correctly.

  I guess it causes this problem.

>  In the 固定幅フォントを選択 - Konsole panel, the bottom part is a text box. I can type or paste text in >  the text box. I↓↓↓↓↓can↓↓↓↓↓paste↓↓↓↓↓five↓↓↓↓↓downarrows, and they display correctly in that >  text box. Konsole corrupts the display in its main window, and the

...

...

>  preview the   corruption.

  Unfortunately, the preview is for font selection dialog and Qt widgets. It is not preview of   Konsole.

  Maybe, the preview should be removed.

  Fuminobu TAKEYAMA

  On 2015/06/12 7:05, Norman Diamond wrote: >  There is a worse bug. > >  In the 固定幅フォントを選択 - Konsole panel, the bottom part is a text box. I can type or paste text in >  the text box. I↓↓↓↓↓can↓↓↓↓↓paste↓↓↓↓↓five↓↓↓↓↓downarrows, and they display correctly in that >  text box. Konsole corrupts the display in its main window, and the

...

...

>  preview the corruption. > >  Yours sincerely,Norman Diamond > > >  Norman Diamond wrote: > >>  I click 設定 > 現在のプロファイルを編集(E)... > 外観 In フォント プレビュー it says Monospace Oh now I see, next >>  to フォントサイズ where it lets me change the size,

...

...

>>  In that panel it says モノスペース instead of Monospace. >> >>  Thank you for the workaround.  I will experiment with some fonts.  I would like to >>  recommend that the preview box should include some full-width characters so that users can >>  judge more easily.  If a font doesn't include glyphs for full-width characters then users >>  will see immediately that they shouldn't experiment with such a font. >> >>  I still think there's a bug. >> >>  Konsole knows that the glyph for 字 is full-width, so Konsole should know that the glyph for >>  downarrow is full-width. >> >>  The version of Konsole that was included in Suse 11.0 did know

...

...

>>  full-width.  I never had to look at what the default font was because it just worked. >> >>  Yours sincerely, Norman Diamond >> >> >>  Fuminobu TAKEYAMA wrote: >> >>>  Ok, Monospace is an alias of IPAGothic on your environment. >>> >>>  The font, IPAGothic provides double-width down-allow glyph although Konsole assumes that >>>  the glyph should have the same width as glyphs such as 'a' and '0'. >>> >>>  A solution is to use another font such as DejaVu Sans Mono. You can change fonts from: >>>  Settings > Edit Current Profile... > Appearance >>> >>>  If you do not need Japanese fonts, please uninstall ipa-gothic-fonts. So another font is >>>  to be selected for Monospace. >>> >>> >>>  Fuminobu TAKEYAMA >>> >>> >>>  On 2015/06/11 6:42, Norman Diamond wrote: >>>>  Yes the font is Monospace.  I did not change it from

opensuse-m17n@opensuse.org Date: 2015/6/13, Sat that downarrow is preview window does not preview window does not there's a clickable button to select a font. that downarrow is the default.  A

...

...

>>>  Konsole settings panel displayed the name Monospace and allowed me to >>  change the >>>  font size but didn't allow me to select a different font. >>>> >>>>  For this e-mail I use Yahoo's web interface from Internet Explorer >> >>>  under Windows, so I translated the output of fc-match -v and fc-match -v Monospace from >>>  UTF-8 to Microsoft's Unicode encoding.  Yahoo's web interface will scramble the text that >>>  I will paste below, so I ALSO attach >>  the >>>  outputs as files. >>>> >>>> >>>>  Output of fc-match -v >>>> >>>>  Pattern has 41 elts (size 48) family: "IPA Pゴシック"(s) "IPAPGothic"(s) familylang: >>>>  "ja"(s) "en"(s) style: "Regular"(w) stylelang: "en"(w) fullname: "IPAPGothic"(w) "IPA P >>>>  ゴシック"(w) fullnamelang: "en"(w)

...

...

"ja"(w) slant: 0(i)(s) weight: 80(i)(s) width:

...

...

>>>>  100(i)(s) size: 12(f)(s) pixelsize: 12.5(f)(s) foundry: "unknown"(w) antialias: >>>>  True(w) hintstyle: 0(i)(w) hinting: True(w) verticallayout: False(s) autohint: True(w) >>>>  globaladvance: True(s) file: "/usr/share/fonts/truetype/ipagp.ttf"(w) index: 0(i)(w) >>>>  outline: True(w) scalable: True(w) dpi: 75(f)(s) rgba: 5(i)(w) scale: 1(f)(s) charset: >>>>  [snip] (w) lang: >>> >>

aa|ay|bg|bi|br|ch|cs|da|de|en|eo|es|et|eu|fi|fj|fo|fur|fy|gd|gl|gv|ho|hu|ia|id|ie|io|is|it|ja|kum|lb|mg|nb|nds|nl|nn|no|nr|nso|oc|om|os|pl|pt|rm|ru|sel|sk|sma|smj|so|sq|ss|st|sv|sw|tk|tl|tn|ts|uz|vo|vot|wa|wen|xh|yap|zu|an|csb|fil|hsb|ht|jv|kj|ku-tr|kwm|li|ms|ng|pap-an|pap-aw|qu|quz|rn|rw|sc|sg|sn|su|za(s)

...

...

...

>>>> >> fontversion: 198574(i)(s) >>>>  capability: "otlayout:hani otlayout:kana

...

...

otlayout:latn"(w) fontformat: "TrueType"(w)

...

...

>>>>  embeddedbitmap: True(w) decorative: False(s) lcdfilter: 0(i)(w) namelang: "ja"(s) >>>>  prgname: "fc-match"(s) hash: >>> >>

"sha256:a63f6153841e56ec9df1b31a54d98d3f41d8aaa1c3d8df30a33502296f5b8068"(w)

...

...

...

>>>>  postscriptname: "IPAPGothic"(w) force_hintstyle: "none"(w) force_autohint: False(w) >>>>  force_bw: False(w) force_bw_monospace: False(w) >>>> >>>>  Output of fc-match -v Monospace >>>> >>>>  Pattern has 42 elts (size 48) family: "IPAゴシック"(s) "IPAGothic"(s) familylang: "ja"(s) >>>>  "en"(s) style: "Regular"(w) stylelang: "en"(w) fullname: "IPAGothic"(w) "IPAゴシック"(w) >>>>  fullnamelang: "en"(w) "ja"(w) slant: 0(i)(s) weight: 80(i)(s) width: 100(i)(s) size: >>>>  12(f)(s) pixelsize: 12.5(f)(s) spacing: 90(i)(w) foundry: "unknown"(w) antialias: >>>>  True(w) hintstyle: 0(i)(w) hinting: True(w) verticallayout: False(s) autohint: True(w) >>>>  globaladvance: True(s) file: "/usr/share/fonts/truetype/ipag.ttf"(w) index: 0(i)(w) >>>>  outline: True(w) scalable: True(w) dpi: 75(f)(s) rgba: 5(i)(w) scale: 1(f)(s) charset: >>>>  [snip] (w) lang: >>> >>

aa|ay|bg|bi|br|ch|cs|da|de|en|eo|es|et|eu|fi|fj|fo|fur|fy|gd|gl|gv|ho|hu|ia|id|ie|io|is|it|ja|kum|lb|mg|nb|nds|nl|nn|no|nr|nso|oc|om|os|pl|pt|rm|ru|sel|sk|sma|smj|so|sq|ss|st|sv|sw|tk|tl|tn|ts|uz|vo|vot|wa|wen|xh|yap|zu|an|csb|fil|hsb|ht|jv|kj|ku-tr|kwm|li|ms|ng|pap-an|pap-aw|qu|quz|rn|rw|sc|sg|sn|su|za(s)

...

...

...

>>>> >> fontversion: 198574(i)(s) >>>>  capability: "otlayout:hani otlayout:kana

...

...

otlayout:latn"(w) fontformat: "TrueType"(w)

...

...

>>>>  embeddedbitmap: False(w) decorative: False(s)

...

...

lcdfilter: 0(i)(w) namelang: "ja"(s)

...

...

>>>>  prgname: "fc-match"(s) hash: >>> >>

...

...

...

>>>>  postscriptname: "IPAGothic"(w) force_hintstyle: "none"(w) force_autohint: False(w) >>>>  force_bw: False(w) force_bw_monospace: False(w) >>>> >>>> >>>> >>>> >>>>> >>>>>  ----- Original Message ----- >>>>>>  From: Fuminobu TAKEYAMA To: opensuse-m17n@opensuse.org Cc: Date: >>>>>>  2015/6/10, Wed 19:43 Subject: Re: [opensuse-m17n] Konsole displays some >>  characters >>>  wrong >>>>>> >>>>>>  Hi, >>>>>> >>>>>>  Which font do you use on Konsole? >>>>>> >>>>>>  Then, can you share the result of fc-match -v? If the font is "Monospace" then run >>>>>>  the following >>>  command: >>>>>>  $ fc-match -v Monospace >>>>>> >>>>>>  # you can snip off "charset:" section >>>>>> >>>>>> >>>>>>  Fuminobu TAKEYAMA >>>>>> >>>>>>  On 2015/06/09 11:41, Norman Diamond wrote: >>>>>>>  In openSUSE 13.2, in Konsole, downarrow↓displays↓wrong. The file encoding, >>>>>>>  Konsole terminal's encoding, and >> >>>  vim's >>>>>  encoding >>>>>>  are all UTF-8. >>>>>>> Five↓↓↓↓↓downarrows↓↓↓↓↓look↓↓↓↓↓like↓↓↓↓↓three. vim counts characters

"sha256:503af4a8b84d1079b8e2e358dc7f7a7fb8cb7a1f212f35eaef6782dbfc75a55e"(w) properly

...

...

...

>>>>>>>  but the display is >>  wrong. >>>>>>>  more, cat, etc., display wrong. >>>>>>> >>>>>>>  I used Suse 11.0 for six years.  Konsole had no problem >>  with >>>>>  downarrows. >>>>>>> >>>>>>>  Yours sincerely, Norman Diamond >>>>>>> >>>>>>  -- To unsubscribe, e-mail: >>  opensuse-m17n+unsubscribe@opensuse.org >>>>>>  To contact the owner, e-mail: >>  opensuse-m17n+owner@opensuse.org >>>>>> >>>>> >>>  -- To unsubscribe, e-mail: opensuse-m17n+unsubscribe@opensuse.org To contact the owner, >>>  e-mail: opensuse-m17n+owner@opensuse.org >>> >>  -- To unsubscribe, e-mail: opensuse-m17n+unsubscribe@opensuse.org To contact the owner, >>  e-mail: opensuse-m17n+owner@opensuse.org >>   -- To unsubscribe, e-mail: opensuse-m17n+unsubscribe@opensuse.org To contact the owner, e-mail:   opensuse-m17n+owner@opensuse.org

-- To unsubscribe, e-mail: opensuse-m17n+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-m17n+owner@opensuse.org

-- To unsubscribe, e-mail: opensuse-m17n+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-m17n+owner@opensuse.org

-- To unsubscribe, e-mail: opensuse-m17n+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-m17n+owner@opensuse.org

-- To unsubscribe, e-mail: opensuse-m17n+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-m17n+owner@opensuse.org

-- To unsubscribe, e-mail: opensuse-m17n+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-m17n+owner@opensuse.org