Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Agreed with what you wrote except "the most backwards compatible (can be passed through many tools intended for ASCII-only)"... That amounts to knowingly sweep bugs under the rug.

If a tool is intended for ascii-only, don't pass it utf-8 strings. Else, you'll probably expose yourself to malformed utf-8 strings and potential problems (e.g. php's mysql_escape vs mysql_real_escape).



Technically, it's not that it can be passed through tools intended for ASCII only. It can be passed through tools which assume an ASCII compatible character set (character set for which ASCII is a subset), which are 8 bit clean, and which don't make incorrect assumptions about being able to truncate strings at arbitrary points and be left with two valid strings.

Which is actually generally true of any tools which had been internationalized with legacy, pre-Unicode character sets like the ISO 8859 series.


Yes, obviously there will be caveats when using a tool beyond its intended domain:

* you do need to know that the tool passes non-ASCII through unchanged

* the text should not contain composed latin glyphs

* you're on your own if you're trimming strings to byte lengths

I've added the second point to my comment.

It's not about sweeping bugs under the rug at all. It's about using non-latin text on the command-line and in code. Most command-line tools are ASCII but will pass through non-ASCII characters unchanged. However, most require that you avoid composed latin glyphs. Since Unicode includes single-codepoint versions of all valid latin accented glyphs and these are the default entry methods, this isn't usually a problem but yes, it really constitutes a subset of UTF-8 and you must know about this limitation to avoid bugs.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: