Unicode

Introduction to Unicode in JavaScript

Hello! You may be familiar with Unicode. This mimics a comprehensive character set within JavaScript code. Unicode is a superior standard that facilitates the representation and handling of text from nearly all writing systems.  Handling strings and text in JavaScript is a significant challenge.

For what does this matter? A wide variety of special characters and symbols from many different languages may be used thanks to Unicode. JavaScript is becoming worldwide. Given the prevalence of text processing in many applications, it is imperative that students studying JavaScript have a solid understanding of Unicode. You are now ready to delve into the more complex Unicode concepts of JavaScript, thanks to this elementary introduction.

Understanding Character Sets and Encodings

Okay time to deconstruct this. Experts call a character set a collection of characters with unique numbers, or "code points." ASCII's 'A' has a code point of 65. Encoding involves transforming code points into binary data, which computers can understand. Additionally, different encodings show characters with different bit counts.

// ASCII encoding of 'A'
01000001

Now, this is where Unicode is utilized. It is a comprehensive character set that encompasses nearly every character from every writing technique that humans have devised worldwide. Unicode is a universal champion because each character is assigned a unique number. Then, we introduce UTF-8, which is the most intriguing method of encoding those Unicode characters. For each character, it employs between one and four 8-bit bytes. This is particularly beneficial for English and other languages, as they contain a variety of intricate characters.

// UTF-8 encoding of 'A'
01000001

// UTF-8 encoding of '€' (Euro sign)
11100010 10000010 10101100

Working with JavaScript text requires understanding these principles. To what purpose? JavaScript's default character set and UTF-8 encoding allow it to handle text in any language, making it ideal for worldwide online applications.

• In character sets, each character has a number.
• Encoding turns integers into binary data.
• The only character set that supports all writing systems is Unicode.
• JavaScript's default encoding is clever and efficient UTF-8.

Mastering these topics will equip you to manipulate text in JavaScript independent of language or symbols.

Difference between ASCII, UTF-8, and Unicode

Time to discuss Unicode, UTF-8, and ASCII. They study how computers engage with text, yet they work differently. American Standard Code for Information Interchange (ASCII) is the first topic. This old character encoding standard provides numerical values to English letters, digits, punctuation marks, and control characters. The basic version uses 7 bits per character to support 128 characters. However, ASCII is quite limited since it only includes English-language characters and no unusual characters from other languages.

// ASCII representation of 'A'
01000001

Unicode is a significant development in the realm of character sets. Its objective is to collect each character from every language in existence and designate it a distinctive code point, akin to a digital identifier. Unicode is the benchmark for universality, boasting a repertoire of more than one million characters.

UTF-8, a flexible character encoding, can now support all Unicode characters. Its ASCII backward compatibility is ingenious. Depending on its nature, each character has one to four 8-bit bytes. Over 95% of online sites utilize UTF-8, the most used character set.

// UTF-8 representation of 'A'
01000001

// UTF-8 representation of '€' (Euro sign)
11100010 10000010 10101100

• ASCII is a character encoding standard that utilizes seven bits to encode English text.
• Unicode contains characters from all languages.
• UTF-8 supports all Unicode characters and is ASCII-compatible.

Why is this topic interesting? Why? Because JavaScript's text processing requires it. JavaScript, which uses UTF-8 and Unicode by default, is a powerful tool for worldwide web development that can handle text in almost any language.
 

How JavaScript handles Unicode

JavaScript processes text using Unicode, allowing it to handle many characters from different languages and symbol sets. Unicode gives each character a unique code point, which JavaScript supports using UTF-16.
Major JavaScript Unicode Handling Issues:

• Encoding UTF-16: A 16-bit coding unit represents characters while most characters fit inside a 16-bit BMP. Other characters, such emojis and unusual scripts, employ surrogate pairs, two 16-bit units.
• Surrogate Pairs and String Length: The length property in JavaScript counts code units, not characters. Characters outside the BMP count as two.
• Code Points: Modern JavaScript supports complete Unicode code points.
• Regular Expressions: The 'u' option makes regex Unicode-aware.

Working with Unicode Strings in JavaScript

JavaScript supports most writing systems by using Unicode characters. JavaScript strings are encoded in UTF-16, where most common characters (in the Basic Multilingual Plane, BMP) use a single 16-bit code unit, whereas emojis and unusual symbols need two surrogate pairs.
Key Ideas and Methods:

• String length: counts code units, not characters. Characters outside BMP count as two.
• Character Access: Direct indexing (str[0]) may mishandle surrogate pairings. Use codePointAt() for precise character retrieval.
• Character creation: String.fromCodePoint() creates characters from code points, including surrogate pairs.
• Regular Expressions using Unicode: Regex should include 'u' for unicode compliance.

Challenges include: 

• Using older methods may make the unicode unstable.
• Older methods include length or charAt().

Unicode Escape Sequences in JavaScript

JavaScript Unicode escape sequences make it easy to incorporate special characters and symbols in strings.

A Unicode escape sequence starts with \u, followed by four hexadecimal digits indicating the character code point (for BMP characters). In current JavaScript, characters outside the BMP are represented as \u{}.

Unicode Escape Sequence Types:

• To represent BMP characters (U+0000 to U+FFFF), use \uXXXX.
• Use surrogate pairs or the contemporary \u{} syntax for characters outside BMP (e.g., emojis) with code points above U+FFFF.

let text = "I \u2764 JavaScript and \u{1F600}!";
console.log(text);

Methods for Manipulating Unicode Characters in JavaScript

JavaScript has various current ways for manipulating Unicode characters, including emojis and non-BMP characters.

• Character Access: charAt() retrieves the character at an index (BMP only). codePointAt() returns the complete Unicode code point for surrogate pairs.
• Creating Characters: String.fromCharCode() creates a character from UTF-16 code units (BMP only).
• String Normalization: normalize() converts different Unicode representations to a consistent form.
• Regex with Unicode: Unicode-Aware Flags use the u flag for accurate matching of Unicode characters.

String.fromCharCode() example:  

console.log(String.fromCharCode(65)); // A

String Normalization example:

let combined = "e\u0301"; // é as 'e' + combining acute accent
console.log(combined.normalize() === "é"); // true

Common Issues with Unicode in JavaScript and How to Solve Them

Unicode in JavaScript allows for different characters and symbols, but its UTF-16 encoding can cause issues, especially with characters beyond the Basic Multilingual Plane. Common challenges and solutions:

• Incorrect String Length: UTF-16 code units, not characters. Emojis and other non-BMP characters employ surrogate pairs, creating erroneous length. Full characters can be handled with Array.from() or for...of.
• Character access: Accessing surrogate pairs with charAt() or indexing yields incomplete characters. Use codePointAt() for correct character retrieval.
• Normalization: Issues with normalization Unicode representations may differ. Normalize() strings before comparison may help.

let a = "e\u0301";
let b = "é";
console.log(a.normalize() === b.normalize()); // true

Using Unicode in Regular Expressions in JavaScript

Regular expressions in JavaScript can handle Unicode characters, but complete compatibility requires extra care, especially for characters beyond the Basic Multilingual Plane. JavaScript's 'u' option makes regular expressions Unicode-aware, properly handling surrogate pairs and other Unicode capabilities.

Features:

• Managing pairs of surrogates: Emojis and other non-BMP characters utilize two UTF-16 code units.
• Case insensitivity: The 'i' flag works with 'u' on Unicode characters.
• Attributes Matching Unicode: Introduced in ES2021, the \p{} syntax matches characters based on script, category, or block properties.
• Code Points: Unicode uses the 'u' flag to implement character escape sequences.

let regex = /ß/i; 
console.log(regex.test("SS")); // false without `u`, true with `u`

Best Practices for Using Unicode in JavaScript

Unicode-based JavaScript provides multilingual characters, symbols, and emojis. But its UTF-16 encoding poses issues, especially with non-BMP characters. Best practices make Unicode processing efficient and error-free.

Practices include:

• Get Code Points Right: Use codePointAt() instead of charAt() for surrogate pairs.
• Normalize Strings: Unicode provides different character representations. Standardize using normalize().
• Carefully Manage String Length: The length attribute counts code units, not characters. Array.from() counts accurately.
• Regular Expression Awareness: Match surrogate pairs and special characters with the 'u' flag.

let str = "e\u0301";
console.log(str.normalize() === "é"); // true