I'm a bit confused about Unicode. It was a repository of linguistic symbols, not raw symbols. More and more it looks like wingdings. Isn't this putting burden on font support and Text processing (what's the lexicographic order of such symbols, using the abstract name ?) ?
They want every symbol used in a document to have a unique encoding, so that you can change fonts without losing meaning. Fonts like wingdings are a horrible hack.
The idea is one (complex) encoding that will represent the info until the end of time. It creates a lot of trouble, but it's still a good idea.
Technically, glyphs are supposed to meet some standards, like being shown in use in running text, before they can be added to unicode. It's not supposed to be a repository of every picture anyone ever dreamed up.
The standards are not applied consistently. Even leaving emoji out of it, the chinese "character" 囍 never occurs in running text, but there it is in unicode.
I don't think it's true that 囍 never occurs in running text - it's used in company names which would be used in text. It would be odd not to have an encoding for such a common character.
All right, I spent some time trying to find the requirement. I did not find it, but my tentative conclusion is that it does not apply to chinese characters.
FROM MEMORY, a while back there was an article on HN complaining that emoji seemed to magically bypass the requirements other characters needed to meet for inclusion in unicode, and that in fact they were commonly in violation. The taco symbol was called out as an example. I can no longer find this article, but it mentioned the running text requirement, and -- I believe -- specifically indicated that use in names does not count as use in running text. (For an idea of why that might be the case, check out http://tvtropes.org/pmwiki/pmwiki.php/Main/LuckyCharmsTitle .)
HOWEVER, I was not even able to find, on the unicode web site, any discussion of a running text requirement at all, for any kind of symbol. Some example proposals do refer to "running text" by name, but they don't indicate why. The example proposal given for adding characters to an existing block ( http://std.dkuug.dk/jtc1/sc2/wg2/docs/n2934.pdf , suggested as a prototype in http://www.unicode.org/faq/char_proposal.html ) does not mention "running text" at all, and doesn't appear to go to much trouble to document it, although some such documentation is given. The rough guidelines for character proposals at http://unicode.org/pending/proposals.html do not refer to "running text" at all, but they do suggest that, late in the process (specifically, on a proposal summary form, which is different from, and subsequent to, an actual proposal), "references to dictionaries and descriptive texts establishing authoritative information" are required.
I conclude that the Unicode standard's preferred criterion for chinese character inclusion is "would an authoritative chinese dictionary include this character", and while the answer to that question for 囍 is not unambiguous -- a lot of dictionaries don't include it -- it's easy to imagine that some do.
I would appreciate a pointer to the actual running text requirements, as well as what they are supposed to apply to, if anyone can provide that.
> It would be odd not to have an encoding for such a common character.
Outside of its use as a wedding decoration, which is plainly nonlinguistic, how common is it?
You're not wrong, but I don't know if it's a great example.
CJK characters are, broadly, an example of the Unicode Consortium trying to be way too reductive about what they'd accept, leading to a lot of bad decisions like Han Unification, which caused a lot of damage and which the Consortium has generally now backed away from and recognized as a bad idea.
So, yes, if you look closely at CJK character sets in Unicode, you can find a lot of decision making that appears to contradict decision making elsewhere in the standard. This is in large part because the decisions they made wrt CJK characters turned out to be largely wrong, and they've since changed their approach.
Unification is a mistake. But 囍 has nothing to do with unification. Do you think that 福倒 should have a code point? Would it be considered one of the chinese characters (very iffy) or one of the holiday symbols?
The 天书 ( https://en.wikipedia.org/wiki/A_Book_from_the_Sky ), by design, consists solely of chinese characters that don't exist. (Theoretically. A couple of them, by oversight, did exist.) They are still recognizably "chinese characters" by virtue of being composed of the same components. Should they have unicode points?
囍 plainly exists, but has no textual use. Is it more similar to 靑 or to ️U+2764 "heavy black heart"?
> Unification is a mistake. But 囍 has nothing to do with unification.
I'm not saying it does, I'm saying Unification illustrates the fact that the Consortium's decision-making with respect to CJK has changed over time, has frequently been illogical, and shouldn't be pointed at as an example of anything good or sane or worthy of precedent.
The fact that 囍 has a code point but 福倒 doesn't have a codepoint is another example of the Consortium being unnecessarily reductive and intransigent about CJK.
> Do you think that 福倒 should have a code point?
Yes. If we want to be able to talk about it in text (like now), I want to be able to encode it in a standardized way.
> Should they have unicode points?
I'd lean towards no, as they're one-offs, not something broader that people want to discuss and use in text. But I'd be ok with adding them, too. We're not running out of space. There's no value in making CJK so much harder to interop with than everything else, in general.
> Yes. If we want to be able to talk about it in text (like now), I want to be able to encode it in a standardized way.
This doesn't make any sense. We talk about things in text by using words, not direct representations. A dog emoji is not necessary or desirable for discussing dogs in text, and a 福倒 emoji is not necessary or desirable for discussing 福倒s in text.
Should the wikipedia page https://en.wikipedia.org/wiki/Statue_of_Liberty be edited to replace the cumbersome phrase "statue of liberty" with the more modern and convenient U+1F5FD 'STATUE OF LIBERTY'?
> A dog emoji is not necessary or desirable for discussing dogs in text, and a 福倒 emoji is not necessary or desirable for discussing 福倒s in text.
"Necessary" is an ill-defined and reductive way of looking at communication. History has shown us that you can't draw bright lines between things you, in the abstract, have decided are the "necessary" subset, and expect the world to follow along.
Linguists have come to understand that you can only describe and follow human, behaviour, not prescribe it.
Anyways, humans plainly found it necessary to annotate their text messages with pictoral indicators of their mood, to the point where it became so widely spread and such a mess that we felt it desirable to standardize the code-point representations. That it isn't desirable in all circumstances or appropriate in all registers of formality does not mean that it isn't an emergent behaviour which will continue to arise whether or not it is "necessary".
tl;dr I don't really give a shit that "dog emoji" isn't appropriate for an academic text on canine surgery. It's more than sufficient to me that it is used millions of times in text messages between regular human beings. Text needn't be formal text to deserve respect in encoding.
> "Necessary" is an ill-defined and reductive way of looking at communication.
I took "If we want to be able to talk about it in text (like now), I want to be able to encode it in a standardized way" as implying that the two clauses were related to each other. Saying "if we want to be able to talk about it" means you're talking about what's necessary for that purpose.
Interesting comments, thanks. It is used in company names, which would make it awkward not to have an encoding for it: there are many characters used just in names in Chinese that would leave locations and people having unencodable names if the characters were not in Unicode.
> there are many characters used just in names in Chinese that would leave locations and people having unencodable names if the characters were not in Unicode
I was under the impression that this describes the current state of affairs, and has since before Unicode came around. I know I've read an article about someone whose 姓 was 马 and whose personal name was a character composed of three 马 stacked left-to-right (which might have been pronounced cheng?) getting harassed because the government couldn't encode the name.
It may be the case, but usage of many of these characters is rarer than 囍. Looking through many of the characters in the cjk Unicode extensions it is not hard to conclude that characters like 福倒 should exist. E.g. 2010f and 20114 which look like 了 and 予 upside down. Or 255d0 which is 石磊 joined together.
The word "running" doesn't appear on that page. (Actually, no requirements at all appear on that page; it speaks strictly in terms of strengthening or weakening the case for inclusion, not disqualifying.) Can you explain briefly why that page is evidence that the running text requirement does not apply to Chinese, and where it specifies what the running text requirement is?
Alternatively, what requirements do apply to Chinese, and would they preclude an invented character like one with 女 on the left and 离 on the right?
I thought the symbol guideline page discussed "running text", but I guess not. Apparently the "running text" requirement isn't part of the published criteria even though it is enforced in discussion.
That character is super common. Way more common than any emoji I'd expect (even teens using emojis won't outweigh the number of weddings and new years in Chinese speaking countries).
Well, even if it doesn't meet unicode inclusion requirements, it is necessary for printing in one of the largest markets in the world. Without that character in unicode, Chinese display systems and printers probably won't use unicode at all (and before unicode they used some standard of their own) - meaning the question is whether unicode wants to be relevant or not, not whether the inclusion requirements fit.
This is used in discussions of the character, would you not consider that text? it does seem to have a more figurative than literal reference than most characters, in a way that I am not sure how to translate into English.
Use of 囍 in discussions of the character 囍 can be reasonably considered nominal use (that is, use within a name). The use of a concrete object to directly represent itself isn't really the same thing as the use of language to refer to a concrete object.
edit: I'd be interested in hearing your thoughts about "it does seem to have a more figurative than literal reference than most characters, in a way that I am not sure how to translate into English", in Chinese if necessary. (No guarantee I'll understand it, but I'm interested.)
In response to your edit, I mean that it has cultural resonance that is unusually strong in relation to its linguistic overtones, in many ways similar to the semantic timbre of a character like 福. The level of abstraction is different from English because of the ideographic nature of characters that means the visual appearance is emphasised, so the boundary that you pick out between reference and referent is more blurred.
I'm not really clear why exactly this character isn't more widely used in text, but I feel this might not be a bright dividing line from more common characters. I think inclusion of the the 福倒 is a harder case to make, but the examples I quoted elsewhere in this thread make me think it should be included. Perhaps not what you were hoping for in terms of elaboration, the problem is more conceptual fuzziness on my side perhaps than language of expression.
Assuming that 囍事 in that passage refers to "a wedding", first I'd admit that that passes pretty much any test of "linguistic use in running text".
Having said that, I note that 喜事 appears in my dictionaries with the gloss "wedding" (well, "any occasion meriting joy, particularly a wedding"), 囍事 does not, and since 囍 is a symbol of weddings which is generally assumed by the Chinese to have the same pronunciation as 喜 it makes for very natural wordplay to substitute it into the word for wedding. I would draw a pretty close analogy with the $ of "Micro$oft" -- it's use in running text, but it shouldn't be taken as evidence that $ is a letter in English.
You just aren't getting it. Do you actually know Chinese, or are you just looking things up in a dictionary?
It is pretty natural to jump from 喜 to 囍 because that is how Chinese works. You take radicals, and you bundle them up. You have the "busho" system where people in the past bundled up little bits and pieces and form new words. No reason why people in the present can't do the same.
Re: Micro$oft being outrageous if $ becomes a part of the alphabet. You are misapplying an English oriented viewpoint. In Chinese, there is no objection to forming words in that way, by incorporating radicals together. It's similar in theme to how in German, you can just keep stringing words together to form larger words. In fact, I actually think in the future, words like Micro$oft should entitle $ to become part of the alphabet! That's a very Chinese way of looking at things.
Language is not static. Systems that try to encode language are descriptive. They can never be prescriptive - otherwise we as a civilization die.
If 囍 wants to be a character point, let it be one. If 福倒 wants to be one, there should be one. Isn't the point of unicode to have enough space to include all these kinds of language artifacts (artifact as in a cultural / historical item thought up by humans) in order so people can uniquely reference each one? They are distinct logical units.
If the unicode rulebooks are too rigid, the rules need to change or the approach needs to change. It's useless to try to argue that xyz character in another language shouldn't/can't be a character - people will just stop using unicode if it doesn't suit their needs.
Reeks of colonialism, that's what it is.
EDIT: as an additional gloss, here's why I think 喜 and 囍 are sometimes used differently, even though by the dictionary definition they seem to be the same. I will explain why I think logically they are different concepts.
喜 is happiness, delight, joy. It is probably an adjective in the English sense (I can't map grammar rules through different languages easily).
事 is an occurrence, an item, something that happens.
When you put them together,
喜事 literally means something happy is happening.
The cultural meaning has turned that into a connotation of "wedding", but it could actually be a ton of happy things. Promotions, and yes - one other really big thing in a person's life: having a baby.
有喜 (means "having happiness") is the traditional way of referring to a woman being pregnant
You can turn that into 家有喜事 - meaning home having something happy - as in this household is having a baby. And you can use it without the 有 - and just use 喜事 to refer to having a baby.
This is different from a wedding.
囍 is a modification of 喜, by doubling up the character and treating it as a radical, people are referring to the idea that there are "two people having happiness" - like a doubled amount of happiness.
In the article linked http://www.chinatimes.com/newspapers/20160623000760-260115 - the 囍事 is used to specifically identify the "wedding" type of 喜事 - it's like trying to avoid the ambiguity and double-entendres that Chinese writing typically embraces and just presents things matter of fact, which is ideal because the article is a newspaper article about customs of towns. Not really something you want people to have multiple interpretations like an essay or a poem, for example.
So logically, there is a difference when trying to use 喜 vs 囍 and I actually really appreciate the author's use of the double version in the text.
I know that not everyone reads these characters in this way, but I do - and I'm sure other people will notice this too. It's the best part of Chinese - not knowing, and not seeing the ambiguity, and one day, someone tells you about it .. and you're like - OMG that's what that means ...
For my earlier indication that this type of character modification is common in chinese:
木 = wood
林 = common last name Lin, also means forest (uncommon on its own)
森 = common character for forest.
The English word "forest" is usually 森林
It's just a doubling and trippling of the 木 radical.
What does it matter that this character is super old - people thousands of years ago thought this up.
Also, if this character weren't so old, would you say that 森 and 林 are both forests and thus don't need separate character points in unicode? That's outrageous!
So now we have a modern version of this modification 喜 -> 囍
And I showed how I think they are different logical concepts.
hmm what's the issue with it being a unicode character point?
POST Edit
In the writing of this post, I think I've come to identify Chinese as an "ambiguity-first" language - I learned Chinese as my mother tongue, but stopped at a elementary school level, and switch over to learning English to a Bachelor's degree level.
In Chinese, puns, double-entendres, and ambiguity just "happens" by default, and you have to work your way to be crystal clear.
English is more straight-forward, with a speaker having to try to make puns or double-entendres.
In the case of 囍, it's a reduction in scope. Modern Chinese people had to create a new word just to narrow down the meaning of 喜 - so that it specifically refers to weddings.
Your whole line of thinking was that 喜 already had meanings inclusive of wedding, so 囍 can't possibly add any more meaning when it also means wedding. In actuality, it took away a bunch of extraneous connotations, and in Chinese, the reduction in complexity is so valuable that it's worth a new word.
I think that it's a mistake to try to over-literate and reduce languages into a set of rulebooks for character encoding - that's all I am trying to put forth - it's best for the person or peoples who speak the language to come up with the encoding for it. I have an elementary school knowledge of Chinese and already I am kinda miffed at why people have an objection to 喜 vs 囍
Imagine how the people who have Bachelor's degrees in Chinese must feel.
In this case, the codepoints were added in part because the proposers could show many printed works (user manuals, I guess) that included sentences such as "to turn the foobar on, press the ■ button", which shows that the glyph between "the" and "button" is in some way like the surrounding glyphs. Chessmen were added for similar reasons, even though very few people actually read either user manuals or chess literature.
The difference between an icon and a letter is small and unclear. & is a symbol but was considered a letter as an example. Chinese characters are words etc.
Good point. Letters.. punctuation.. symbol .. the lines are blurry. If I may I'd say that & is a symbol that represent a grammar connective. Which is a generic abstraction and won't cause explosion like having symbols for every word out there.
We may think that we are enlightened beings but the fact is that pictures comprise a lot of how we communicate now and in the past. Are emojis that different from hieroglyphics?
Last I checked, Unicode don't actually have anything like coverage of the entirety of every script and alphabet. On the other hand, approving emoji and random icons delights Westerners.
You don't have to point to human history, though that's a good source of missing scripts. Waving off scripts actually in use as "increasingly obscure", while cheering Unicode throwing in any icon random geeks pitch to them, misses the purpose of Unicode.
You're tossing that assertion around without supporting it – what commonly used characters are not in Unicode? How many people use them? Are they not in Unicode because nobody cares or because there is a lack of someone authoritative helping codify the list or contentious disagreements about some aspects of that work?
Seriously, you're able to Google up those other links, but you somehow can't find the (non-exhaustive) Unsupported Scripts list or the Proposed New Scripts pages on the Unicode site? And, without knowing the situation for any of them, you're going to throw out excuses for why the absences don't matter?
These aren't characters, but entire scripts that are not part of the standard. Nor are major scripts like kanji complete.
Again, you're the one making the claim. Can you precisely state what you believe to be the problem and cite some sources that this is a major problem and that nobody is working on it?
More importantly, ask why it seems unreasonable that a small number of very widely-used ISO standard symbols were incorporated quickly? Wouldn't that be the most reasonable expectation since it lacks the political heat of e.g. Han unification and doesn't require any research or debate to establish that they are used, have a precise meaning, and are not covered by existing codepoints?
I've already stated my complaint: that getting gratuitous icons into Unicode is easier than actual scripts for human languages. Since you're having Google issues, I'll link you to a page I already mentioned, which it self links to other relevant pages http://unicode.org/standard/unsupported.html
I love the echoing nature of these counter-arguments, that a problem doesn't even exist unless it's "major" and "nobody is working on it". I wonder how many actual different human beings have responded to me in this thread...
You might find your conversations work better if you respond to what people are actually saying rather than repeating yourself or assuming that other people don't know how to use Google, particularly after they've already sent you links which comprehensively disprove your assertion by demonstrating how many new characters are being added and that emoji constitute less than 1% of the 7,500 new characters in Unicode 9.0.
Your original claim was that “Unicode don't actually have anything like coverage of the entirety of every script and alphabet” but you're arguing about things which affect something like 0.008% of people – not even their primary usage – and for which there is work in progress to support!
Nobody is saying that Unicode is complete, but like any other human effort there's a limited amount of time to work on things. At some point things which are used daily by billions of people are going to get prioritized over things which are used infrequently by thousands of people, and it's hard to argue that this is wrong even if you – like me – want to have 100% of human language represented in Unicode.
"You might find your conversations work better if you respond to what people are actually saying rather than repeating yourself"
You're going to seriously say that after your last few posts? Two posts into this exchange, you moved the goalposts, and you hammered that button repeatedly.
But at least you actually looked at the proof you repeatedly demanded, even I had already mentioned the pages. You didn't bother reading much of it, or to note that goes well beyond a couple scripts on that page to other incomplete scripts and as-yet entirely unimplemented scripts. But you at least made that minimum effort.
And the limited time to work on these things is exactly the issue. There are scripts not yet in the standard and major language scripts that aren't complete - but we've got "pile of poo" and a slew of emoji. And now, we've got four power button icons that a handful of people demanded.
> You're going to seriously say that after your last few posts? Two posts into this exchange, you moved the goalposts, and you hammered that button repeatedly.
You started this conversation with “Unicode don't actually have anything like coverage of the entirety of every script and alphabet.” It's hardly moving the goalposts to question how complete Unicode has to be to qualify as “anything like” or how much weight usage should have.
> But at least you actually looked at the proof you repeatedly demanded, even I had already mentioned the pages. You didn't bother reading much of it, or to note that goes well beyond a couple scripts on that page to other incomplete scripts and as-yet entirely unimplemented scripts. But you at least made that minimum effort.
Before you could call that proof, you have to clearly articulate the questions it could answer. Note that my first comment indicated a clear understanding of how Unicode works – the process is not in question here, only the thresholds you haven't articulated. All I've been trying to get you to state is precisely what your rules would be for coverage of human languages before we can add anything else and how much usage should factor into that. There's also a much harder question of trying to come up with a rule which to say why a pictograph, the phaistos disc symbols, etc. are valid for inclusion but a modern symbol used millions of times a day around the world to communicate is not?
While thinking about this, it's also worth remembering that despite your apparent belief that emoji are a Western novelty, the question was how to improve Unicode adoption in Japan and that required having an answer for the millions of people who were using systems which relied on non-standard encodings and by most accounts Japanese carriers were resistant to adopting Unicode without having a standard to replace those ad-hoc systems. I think that decision should have been handled differently (i.e. assigning an emoji plane) but it was driven by understandable technical reasons affecting large numbers of people on a daily basis. Since that decision was made, the additional cost to add a small number of non-controversial additions which do not require scholarly research or documentation does not seem excessive — we are, after all, talking about a small percentage of the new symbols in Unicode 9.0.
