Comments on: Why Unicode is subtle http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/ The blog of John D. Cook Sat, 11 Feb 2012 01:10:06 -0500 http://wordpress.org/?v=2.8.4 hourly 1 By: John http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/comment-page-1/#comment-118329 John Wed, 30 Nov 2011 03:21:44 +0000 http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/#comment-118329 I don't know anything about the J language, but here's their web site: http://www.jsoftware.com/ Also, Tracy Harms (@kaleidic on Twitter) often writes about J. I don’t know anything about the J language, but here’s their web site: http://www.jsoftware.com/

Also, Tracy Harms (@kaleidic on Twitter) often writes about J.

]]> By: human mathematics http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/comment-page-1/#comment-118294 human mathematics Tue, 29 Nov 2011 23:54:35 +0000 http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/#comment-118294 John, thank you for sharing your Unicode wisdom. Since you've now become an expert on the subject in my mind, I'm coming to you with a question. I'm just starting to play with the <code>J</code> language. As a derivative of APL, <code>J</code> doesn't always stick to ASCII. For example if you run <code>a:</code> in the <code>J</code> console, you get the following list of characters which comprise its alphabet:  ┌┬┐├┼┤└┴┘│─ !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~������������������������������������������������������������������������������������������������������������������������������ If you know of any J-specific unicode resources, would you mind adding them to your list? John, thank you for sharing your Unicode wisdom. Since you’ve now become an expert on the subject in my mind, I’m coming to you with a question. I’m just starting to play with the J language.

As a derivative of APL, J doesn’t always stick to ASCII. For example if you run a: in the J console, you get the following list of characters which comprise its alphabet:



┌┬┐├┼┤└┴┘│─ !”#$%&’()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~������������������������������������������������������������������������������������������������������������������������������

If you know of any J-specific unicode resources, would you mind adding them to your list?

]]> By: Chris Charabaruk http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/comment-page-1/#comment-114834 Chris Charabaruk Mon, 14 Nov 2011 23:41:24 +0000 http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/#comment-114834 Marius: Because in most cases, you can simply treat UTF-16 as UCS-2, since the need for mapping into 0x1???? points rarely comes up for most applications. As for endianness, that's what the BOM bytes are for, if present. If not, just treat as system-default endian and user beware. Marius: Because in most cases, you can simply treat UTF-16 as UCS-2, since the need for mapping into 0×1???? points rarely comes up for most applications. As for endianness, that’s what the BOM bytes are for, if present. If not, just treat as system-default endian and user beware.

]]> By: Marius Gedminas http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/comment-page-1/#comment-114373 Marius Gedminas Sat, 12 Nov 2011 14:37:01 +0000 http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/#comment-114373 UTF-16 simpler to process? How so? It combines the disadvantages of UTF-8 (variable-length character encoding) with the disadvantages of UTF-32 (endianness issues). UTF-16 simpler to process? How so? It combines the disadvantages of UTF-8 (variable-length character encoding) with the disadvantages of UTF-32 (endianness issues).

]]> By: Draw a symbol, look it up — The Endeavour http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/comment-page-1/#comment-114369 Draw a symbol, look it up — The Endeavour Sat, 12 Nov 2011 14:18:42 +0000 http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/#comment-114369 [...] Why Unicode is subtle The disappointing state of Unicode fonts Entering Unicode characters in Windows and Linux Inserting graphics in Twitter messages [...] [...] Why Unicode is subtle The disappointing state of Unicode fonts Entering Unicode characters in Windows and Linux Inserting graphics in Twitter messages [...]

]]> By: Johannes http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/comment-page-1/#comment-40871 Johannes Mon, 28 Jun 2010 23:23:20 +0000 http://www.johndcook.com/blog/2008/04/05/why-unicode-is-subtle/#comment-40871 Actually, Unicode is a 21-bit character set. Then, the question about σ and ς is probably a hsitorical one. Unicode is committed on a 1:1 mapping of legacy character sets into Unicode. So every character that was once included in a character set has a corresponding code point in the UCS. This isn't nice from an idealist point of view but Unicode is essentially a tool that tries to work in the current (non-ideal) world. Another thing to note about Unicode is that it's very complex as a whole. The character set as such is only a tiny part. The standard also includes collation rules for various languages, algorithms for bidirectional text display and handling and many more things. Another point that's making dealing with it so complex is that the application working with the raw data is one thing, the proper rendering and fonts are a different beast. Latin is one of the easiest scripts to support and yet even there things can go horribly wrong. A layout engine (such as Uniscribe in Windows) has to to proper ligatures of characters in Indic or Arabic scripts, contextual glyphs are needed in the latter as well. Diacritical marks and other combining characters must be positioned properly, mixing writing directions due to script changes is also a fairly complex matter. Rules for embedding sinographs and Latin into Mongolic script (all of them use a different writing direction) exist. Things like those make the standard quite complex to begin with – a necessary but unfortunate consequence of supporting every written language that exists (or existed). Actually, Unicode is a 21-bit character set.

Then, the question about σ and ς is probably a hsitorical one. Unicode is committed on a 1:1 mapping of legacy character sets into Unicode. So every character that was once included in a character set has a corresponding code point in the UCS. This isn’t nice from an idealist point of view but Unicode is essentially a tool that tries to work in the current (non-ideal) world.

Another thing to note about Unicode is that it’s very complex as a whole. The character set as such is only a tiny part. The standard also includes collation rules for various languages, algorithms for bidirectional text display and handling and many more things.

Another point that’s making dealing with it so complex is that the application working with the raw data is one thing, the proper rendering and fonts are a different beast. Latin is one of the easiest scripts to support and yet even there things can go horribly wrong. A layout engine (such as Uniscribe in Windows) has to to proper ligatures of characters in Indic or Arabic scripts, contextual glyphs are needed in the latter as well. Diacritical marks and other combining characters must be positioned properly, mixing writing directions due to script changes is also a fairly complex matter. Rules for embedding sinographs and Latin into Mongolic script (all of them use a different writing direction) exist. Things like those make the standard quite complex to begin with – a necessary but unfortunate consequence of supporting every written language that exists (or existed).

]]>