{"uuid": "6f678f70-9592-4355-a98c-c19db91cae1f", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2021-42574", "type": "seen", "source": "https://gist.github.com/StructSeeker/c3edbde498d99374236f0ddac5037f0e", "content": "\n\n\n    \n    \n    Advanced Unicode: Technical Reference for Programmers\n    \n        :root {\n            --bg: #f8f9fa;\n            --surface: #ffffff;\n            --text: #1a1a2e;\n            --text-muted: #4a4a6a;\n            --border: #e0e0e8;\n            --accent: #2d6cdf;\n            --accent-light: #e8f0fe;\n            --code-bg: #f0f2f5;\n            --code-text: #1a1a2e;\n            --table-header: #2d6cdf;\n            --table-header-text: #ffffff;\n            --table-stripe: #f4f6fa;\n            --warning-bg: #fff8e1;\n            --warning-border: #f0c040;\n            --danger-bg: #fce8e8;\n            --danger-border: #d94444;\n            --success-bg: #e8f5e9;\n            --success-border: #43a047;\n            --radius: 10px;\n            --shadow: 0 2px 12px rgba(0, 0, 0, 0.06);\n        }\n\n        * {\n            box-sizing: border-box;\n            margin: 0;\n            padding: 0;\n        }\n\n        body {\n            font-family: 'Segoe UI', system-ui, -apple-system, BlinkMacSystemFont, 'Helvetica Neue', Arial, sans-serif;\n            background: var(--bg);\n            color: var(--text);\n            line-height: 1.75;\n            padding: 2rem 1rem;\n            font-size: 16px;\n        }\n\n        .container {\n            max-width: 1100px;\n            margin: 0 auto;\n            background: var(--surface);\n            padding: 2.5rem 3rem;\n            border-radius: var(--radius);\n            box-shadow: var(--shadow);\n        }\n\n        @media (max-width: 768px) {\n            .container {\n                padding: 1.5rem 1.25rem;\n            }\n            body {\n                padding: 1rem 0.5rem;\n                font-size: 15px;\n            }\n        }\n\n        h1 {\n            font-size: 2.2rem;\n            font-weight: 700;\n            color: var(--text);\n            border-bottom: 3px solid var(--accent);\n            padding-bottom: 0.75rem;\n            margin-bottom: 1.5rem;\n            letter-spacing: -0.02em;\n        }\n\n        h2 {\n            font-size: 1.6rem;\n            font-weight: 650;\n            color: var(--accent);\n            margin-top: 2.8rem;\n            margin-bottom: 1rem;\n            padding-bottom: 0.4rem;\n            border-bottom: 1px solid var(--border);\n        }\n\n        h3 {\n            font-size: 1.25rem;\n            font-weight: 600;\n            color: var(--text);\n            margin-top: 2rem;\n            margin-bottom: 0.75rem;\n        }\n\n        h4 {\n            font-size: 1.05rem;\n            font-weight: 600;\n            color: var(--text-muted);\n            margin-top: 1.5rem;\n            margin-bottom: 0.5rem;\n            text-transform: uppercase;\n            letter-spacing: 0.04em;\n        }\n\n        p {\n            margin-bottom: 1rem;\n            color: var(--text);\n        }\n\n        ul, ol {\n            margin-bottom: 1rem;\n            padding-left: 1.8rem;\n        }\n\n        li {\n            margin-bottom: 0.4rem;\n        }\n\n        li > ul, li > ol {\n            margin-top: 0.3rem;\n            margin-bottom: 0.3rem;\n        }\n\n        code {\n            font-family: 'Cascadia Code', 'Fira Code', 'JetBrains Mono', 'SF Mono', Consolas, monospace;\n            background: var(--code-bg);\n            padding: 0.15em 0.45em;\n            border-radius: 4px;\n            font-size: 0.9em;\n            color: var(--code-text);\n            white-space: nowrap;\n        }\n\n        pre {\n            background: #1e1e2e;\n            color: #cdd6f4;\n            padding: 1.2rem 1.5rem;\n            border-radius: 8px;\n            overflow-x: auto;\n            margin-bottom: 1.2rem;\n            font-size: 0.88rem;\n            line-height: 1.6;\n            box-shadow: inset 0 1px 4px rgba(0,0,0,0.2);\n        }\n\n        pre code {\n            background: none;\n            padding: 0;\n            color: inherit;\n            font-size: inherit;\n            white-space: pre;\n        }\n\n        .cp {\n            font-family: 'Cascadia Code', 'Fira Code', monospace;\n            background: var(--accent-light);\n            color: var(--accent);\n            padding: 0.1em 0.4em;\n            border-radius: 4px;\n            font-size: 0.85em;\n            font-weight: 600;\n            white-space: nowrap;\n        }\n\n        .char-demo {\n            font-size: 1.4em;\n            vertical-align: middle;\n            line-height: 1;\n        }\n\n        table {\n            width: 100%;\n            border-collapse: collapse;\n            margin-bottom: 1.5rem;\n            font-size: 0.92rem;\n            border-radius: 8px;\n            overflow: hidden;\n            box-shadow: 0 1px 6px rgba(0,0,0,0.05);\n        }\n\n        thead th {\n            background: var(--table-header);\n            color: var(--table-header-text);\n            padding: 0.7rem 1rem;\n            text-align: left;\n            font-weight: 600;\n            font-size: 0.85rem;\n            text-transform: uppercase;\n            letter-spacing: 0.04em;\n        }\n\n        tbody td {\n            padding: 0.65rem 1rem;\n            border-bottom: 1px solid var(--border);\n            vertical-align: top;\n        }\n\n        tbody tr:nth-child(even) {\n            background: var(--table-stripe);\n        }\n\n        tbody tr:hover {\n            background: var(--accent-light);\n            transition: background 0.15s;\n        }\n\n        .callout {\n            padding: 1rem 1.25rem;\n            border-radius: 8px;\n            margin-bottom: 1.2rem;\n            border-left: 4px solid;\n        }\n\n        .callout-warning {\n            background: var(--warning-bg);\n            border-color: var(--warning-border);\n        }\n\n        .callout-danger {\n            background: var(--danger-bg);\n            border-color: var(--danger-border);\n        }\n\n        .callout-success {\n            background: var(--success-bg);\n            border-color: var(--success-border);\n        }\n\n        .callout-info {\n            background: var(--accent-light);\n            border-color: var(--accent);\n        }\n\n        .callout strong {\n            display: block;\n            margin-bottom: 0.3rem;\n        }\n\n        .toc {\n            background: var(--bg);\n            border: 1px solid var(--border);\n            border-radius: 8px;\n            padding: 1.25rem 1.5rem;\n            margin-bottom: 2rem;\n        }\n\n        .toc h3 {\n            margin-top: 0;\n            margin-bottom: 0.75rem;\n            font-size: 1.1rem;\n        }\n\n        .toc ol {\n            margin-bottom: 0;\n            padding-left: 1.5rem;\n        }\n\n        .toc li {\n            margin-bottom: 0.25rem;\n        }\n\n        .toc a {\n            color: var(--accent);\n            text-decoration: none;\n            font-weight: 500;\n        }\n\n        .toc a:hover {\n            text-decoration: underline;\n        }\n\n        .badge {\n            display: inline-block;\n            padding: 0.15em 0.6em;\n            font-size: 0.75rem;\n            font-weight: 700;\n            border-radius: 12px;\n            text-transform: uppercase;\n            letter-spacing: 0.05em;\n            vertical-align: middle;\n            margin-left: 0.4em;\n        }\n\n        .badge-deprecated {\n            background: #ffe0e0;\n            color: #c62828;\n        }\n\n        .badge-recommended {\n            background: #e0f2e0;\n            color: #2e7d32;\n        }\n\n        .badge-critical {\n            background: #ffebee;\n            color: #b71c1c;\n        }\n\n        .badge-info {\n            background: #e3f2fd;\n            color: #1565c0;\n        }\n\n        hr {\n            border: none;\n            border-top: 1px solid var(--border);\n            margin: 2rem 0;\n        }\n\n        .source-list {\n            font-size: 0.88rem;\n            color: var(--text-muted);\n        }\n\n        .source-list li {\n            margin-bottom: 0.3rem;\n        }\n\n        .source-list a {\n            color: var(--accent);\n            text-decoration: none;\n        }\n\n        .source-list a:hover {\n            text-decoration: underline;\n        }\n\n        .kbd {\n            display: inline-block;\n            padding: 0.1em 0.5em;\n            font-size: 0.8em;\n            font-family: monospace;\n            background: #eee;\n            border: 1px solid #ccc;\n            border-radius: 4px;\n            box-shadow: 0 1px 0 #bbb;\n        }\n\n        .mono-block {\n            font-family: 'Cascadia Code', 'Fira Code', monospace;\n            background: var(--code-bg);\n            padding: 0.75rem 1rem;\n            border-radius: 6px;\n            font-size: 0.88rem;\n            line-height: 1.7;\n            margin-bottom: 1rem;\n            overflow-x: auto;\n        }\n    \n\n\n\n\n\n    \nAdvanced Unicode: A Comprehensive Technical Reference\n\n    \n\n        \nTable of Contents\n        \n\n            \nBehavioral & Semantic Concepts\n                \n\n                    \nCase Mapping\n                    \nCollation\n                    \nCompatibility Decompositions & Confusables\n                    \nUnicode-Aware Regexes\n                    \nBidirectional Text (BiDi)\n                \n            \n            \nImplementation & Optimization\n                \n\n                    \nSparse Code Point Storage\n                    \nUTF-8 Decoding Optimization\n                    \nOptimized String Comparison & NFC Normalization\n                \n            \n            \nUnicode Blocks & Planes\n                \n\n                    \nThe 17 Planes\n                    \nFunctional Categorization of Blocks\n                    \nBlock vs. Script Property\n                \n            \n            \nUnicode Tag Characters\n            \nUnicode Character Properties (UCD)\n            \nLibraries & Databases\n            \nGeneral Category Examples\n            \nUnicode for Language/DSL Developers\n                \n\n                    \nIdentifiers: XID_Start & XID_Continue\n                    \nLexical Disjointness\n                    \nNormalization & Comparison\n                    \nSecurity & Confusables (UTS #39)\n                    \nComments & BiDi Attacks\n                \n            \n            \nMath Literals, Operators & Keywords\n            \nUAX #34 Relevance\n            \nSources\n        \n    \n\n    \n    \n1. Behavioral & Semantic Concepts\n\n    \n1.1 Case Mapping\n    \nBeyond simple 1:1 ASCII conversions (a \u2192 A), Unicode case mapping involves complex rules:\n    \n\n        \nLength-changing mappings: Some characters change length when cased. German \u00df (U+00DF) becomes SS in uppercase.\n        \nContext-sensitive: Greek final sigma (\u03c3 vs. \u03c2).\n        \nLocale-sensitive: Turkish i uppercases to \u0130 (dotted I, U+0130) and I lowercases to \u0131 (dotless i, U+0131).\n    \n    \n\n        Implementation Note\n        Use Simple_Uppercase_Mapping / Simple_Lowercase_Mapping for 1:1 mappings, and Uppercase_Mapping / Lowercase_Mapping for 1-to-many mappings.\n    \n\n    \n1.2 Collation\n    \nSorting strings alphabetically is non-trivial across languages. The Unicode Collation Algorithm (UCA) uses multi-level weight keys:\n    \n\n        \n            LevelDifferenceExample\n        \n        \n            PrimaryBase lettera vs. b\n            SecondaryAccenta vs. \u00e1\n            TertiaryCase / widtha vs. A\n        \n    \n    \nLanguage-specific rules further complicate matters. For example, \u00e5 sorts after z in Swedish, but with a in German.\n\n    \n1.3 Compatibility Decompositions & Confusables\n\n    \nCompatibility Decomposition (NFKD / NFKC)\n    \nMaps visually or semantically distinct variants to standard base characters:\n    \n\n        \n\u00b2 (U+00B2) \u2192 2\n        \n\uff21 (U+FF21) \u2192 A\n        \n\ufb01 (U+FB01) \u2192 fi\n        \n\u2460 (U+2460) \u2192 1\n        \n\u00bd (U+00BD) \u2192 1/2\n    \n\n    \n\n        \u26a0\ufe0f Semantic Destruction\n        Running NFKC on mathematical or technical text collapses structural meaning:\n        \n\n            \n                OriginalMeaningPost-NFKCResulting Meaning\n            \n            \n                10\u00b2Ten squared (100)102One hundred two\n                x\u00b2 + y\u00b2Algebraic equationx2 + y2Variable names x2 and y2\n                m\u00b2Square metersm2Malformed text / ambiguous\n            \n        \n    \n\n    \nWhy does Unicode have compatibility decompositions? They exist for legacy interoperability\u2014converting old character sets (e.g., IBM or Japanese encodings) into Unicode without data loss. NFKD/NFKC was designed for:\n    \n\n        \nFuzzy search & indexing: Searching \"m2\" returns documents with \"m\u00b2\".\n        \nIdentifier folding: Preventing visually confusing accounts like admin and \uff41\uff44\uff4d\uff49\uff4e.\n    \n\n    \nConfusables\n    \nCharacters from different scripts that look identical or visually similar (e.g., Latin A vs. Cyrillic \u0410). Critical for security to prevent homograph attacks in domains and usernames.\n\n    \n1.4 Unicode-Aware Regexes\n    \nStandard regex operates on bytes or ASCII ranges. Unicode regexes support character properties:\n    \n\n        \n\\p{Arabic} \u2014 match by script\n        \n\\p{Lu} \u2014 uppercase letters\n        \n\\X \u2014 grapheme cluster boundaries (instead of isolated code units)\n    \n\n    \n1.5 Bidirectional Text (BiDi)\n    \nHandles mixing left-to-right (LTR) scripts like English with right-to-left (RTL) scripts like Arabic or Hebrew. The Unicode BiDi algorithm assigns directional types to every character and resolves embedding levels to render mixed strings correctly.\n\n    \n    \n2. Implementation & Optimization Challenges\n\n    \n2.1 Sparse Code Point Storage\n    \nUnicode spans up to U+10FFFF (1,114,112 possible code points), but vast regions are unassigned. Storing properties in a flat array wastes memory. Implementations use:\n    \n\n        \nMulti-level lookup tables: 2-stage or 3-stage arrays.\n        \nRadix trees / tries: O(1) lookups with minimal memory.\n    \n\n    \n2.2 UTF-8 Decoding Optimization\n    \nNaive UTF-8 decoders check branch conditions byte-by-byte, causing high branch misprediction rates. Modern high-performance decoders use:\n    \n\n        \nState machines: Reduce branching.\n        \nSIMD vectorization: Libraries like simdutf validate and decode multiple UTF-8 bytes in parallel.\n    \n\n    \n2.3 Optimized String Comparison & NFC Normalization\n    \n\n        \nFast-path comparison: Raw byte-level comparison first; fall back to full UCA collation weights only when bytes differ.\n        \nNFC normalization: Pre-composed \u00e9 (U+00E9) vs. e + combining accent (U+0065 U+0301). Quick-check algorithms scan to verify if a string is already normalized before allocating memory to mutate it.\n    \n\n    \n    \n3. Unicode Blocks & Planes\n\n    \nA Unicode Block is a named, continuous range of code points grouping related characters. Every code point belongs to exactly one block (or No_Block). Block boundaries are fixed at multiples of 16 (ending in ...0 to ...F).\n\n    \n3.1 Macro-Level: The 17 Planes\n    \n\n        \n            PlaneNameRangePrimary Block Contents\n        \n        \n            0BMP (Basic Multilingual Plane)U+0000 \u2013 U+FFFFLatin, Cyrillic, Greek, Arabic, CJK Unified Ideographs, basic math, punctuation, surrogates. Almost all modern scripts.\n            1SMP (Supplementary Multilingual Plane)U+10000 \u2013 U+1FFFFHistoric scripts (Linear B, Hieroglyphs), specialized math symbols, musical notation, emojis.\n            2SIP (Supplementary Ideographic Plane)U+20000 \u2013 U+2FFFFRare/Historic CJK Unified Ideographs (Ext. B\u2013F).\n            3TIP (Tertiary Ideographic Plane)U+30000 \u2013 U+3FFFFCJK Ext. G, H, I, ancient seal/oracle scripts.\n            4\u201313Unassigned PlanesU+40000 \u2013 U+DFFFFReserved for future expansion.\n            14SSP (Supplementary Special-purpose Plane)U+E0000 \u2013 U+EFFFFNon-graphical syntax, tag characters, variation selectors.\n            15\u201316PUA (Private Use Areas)U+F0000 \u2013 U+10FFFFReserved for private font makers and custom encoding schemes.\n        \n    \n\n    \n3.2 Functional Categorization of Blocks\n    \n\n        \nStandard Scripts & Alphabets: Basic Latin, Cyrillic, Devanagari, Arabic. Supplements/Extensions when a block runs out of room (e.g., Latin Extended-A, Arabic Extended-A).\n        \nCJK Unified Ideographs: Split across massive blocks (20,992 characters in the BMP alone; additional across Planes 2 and 3).\n        \nCombining Marks: Strictly for diacritics and accents (e.g., Combining Diacritical Marks).\n        \nSymbols & Technical Characters: Mathematical Operators, Box Drawing, Currency Symbols, Dingbats, Emojis.\n        \nPunctuation & Layout Controls: General Punctuation, Supplemental Punctuation.\n        \nSurrogates & Special Code Points: High/Low Surrogates (U+D800\u2013U+DFFF for UTF-16), Specials (BOM, invalid characters).\n    \n\n    \n3.3 Important Distinction: Block vs. Script Property\n    \n\n        \n            ConceptDefinitionExample\n        \n        \n            BlockContiguous memory boundary set when standard was written.U+0061 (a) is in Basic_Latin.\n            ScriptLinguistic classification of the character.Latin letters in Currency Symbols block (U+20A1) have Script = Latin.\n        \n    \n\n    \n    \n4. Unicode Tag Characters\n\n    \nUnicode Tag Characters are 97 invisible control characters in the Tags block (U+E0000 \u2013 U+E007F) within Plane 14. Originally added to embed metadata (language tagging) directly into plain text.\n\n    \n\n        \n            ConceptASCII RangeTag Character RangeDescription\n        \n        \n            Language Tag (Start)N/AU+E0001Signals start of tag sequence\n            Tag ASCII Space\u2013TildeU+0020 \u2013 U+007EU+E0020 \u2013 U+E007EInvisible clones of ASCII characters\n            Cancel Tag (End)N/AU+E007FTerminates tag sequence\n        \n    \n\n    \n\n        \u26a0\ufe0f Deprecated for Language Tagging\n        The Unicode Consortium officially deprecated language tag characters due to security risks (invisible bytes, homograph-style hidden payloads) and duplication of higher-level protocols (HTML lang attributes, JSON fields).\n    \n\n    \nModern Exception: Subdivision Emoji Flags\n    \nTag characters were recycled for Emoji Tag Sequences (UTS #51). Regional sub-entities (Scotland gbsco, England gbeng, California usca) lack 2-letter ISO 3166-1 codes.\n    \nExample \u2014 Flag of Scotland (\ud83c\udff4\udb40\udc67\udb40\udc62\udb40\udc73\udb40\udc63\udb40\udc74\udb40\udc7f):\n    \n\n        Base Emoji: \ud83c\udff4 (U+1F3F4 Black Flag)\n        ISO 3166-2 Code in Tags:\n          U+E0067 (Tag Latin Small Letter G)\n          U+E0062 (Tag Latin Small Letter B)\n          U+E0073 (Tag Latin Small Letter S)\n          U+E0063 (Tag Latin Small Letter C)\n          U+E006F (Tag Latin Small Letter O)\n        Terminator: U+E007F (Cancel Tag)\n    \n    \nWhen an emoji rendering engine encounters this exact sequence, it glues the invisible tags to the base flag and displays the single Saltire flag icon.\n\n    \n    \n5. Unicode Character Properties (UCD)\n\n    \nEvery code point has standardized properties defined in the Unicode Character Database (UCD). These determine behavior during rendering, layout, parsing, regex matching, and line breaking.\n\n    \n5.1 Script Property (and Script_Extensions)\n    \nScript (sc) assigns a code point to a specific writing system (Latin, Cyrillic, Han, Arabic).\n    \n\n        \nCommon (Zyyy): Digits, spaces, punctuation used across scripts.\n        \nInherited (Zinh): Combining marks that inherit the base character's script.\n        \nScript_Extensions (scx): Handles characters belonging to multiple scripts (e.g., shared punctuation used in both Han and Katakana).\n    \n\n    \n5.2 General Category (gc)\n    \n\n        \n            Major ClassCode & NameSubcategoriesExamples\n        \n        \n            Letter (L)Lu, Ll, Lt, Lm, LoUppercase, Lowercase, Titlecase, Modifier, OtherA (Lu), a (Ll), \u6f22 (Lo)\n            Mark (M)Mn, Mc, MeNonspacing, Spacing combining, Enclosing\u25cc\u0301 (Mn), \u25cc\u20e0 (Me)\n            Number (N)Nd, Nl, NoDecimal digit, Letterlike, Other5 (Nd), \u2166 (Nl), \u00bd (No)\n            Punctuation (P)Pc, Pd, Ps, Pe, PoConnector, Dash, Open, Close, Other_ (Pc), - (Pd), ( (Ps), ! (Po)\n            Symbol (S)Sm, Sc, Sk, SoMath, Currency, Modifier, Other+ (Sm), $ (Sc), \ud83d\ude00 (So)\n            Separator (Z)Zs, Zl, ZpSpace, Line, ParagraphStandard space (Zs)\n            Other (C)Cc, Cf, Cs, Co, CnControl, Format, Surrogate, Private Use, Unassigned\\n (Cc), ZWJ (Cf)\n        \n    \n\n    \n5.3 Binary (Boolean) Properties\n    \n\n        \nAlphabetic (Alpha): Letters plus letter-like numbers/symbols.\n        \nUppercase / Lowercase: Fast binary checks.\n        \nDefault_Ignorable_Code_Point (DI): Characters to be ignored by renderers (e.g., soft hyphens, hidden tags).\n        \nEmoji / Emoji_Presentation / Emoji_Modifier_Base: Controls emoji rendering and skin-tone modifiers.\n        \nWhite_Space: All space and layout-separating characters.\n    \n\n    \n5.4 Directional & Rendering Properties\n    \n\n        \nBidi Class (bc): Left_To_Right, Right_To_Left, European_Number, etc.\n        \nCombining Class (ccc): Sequences overlapping diacritics deterministically. Non-combining = 0; accents get specific integer values.\n        \nGrapheme Cluster Break (GCB): Dictates user-perceived character counting.\n    \n\n    \n5.5 Casing Properties\n    \n\n        \nSimple_Uppercase_Mapping / Simple_Lowercase_Mapping: 1-to-1 mappings.\n        \nUppercase_Mapping / Lowercase_Mapping: 1-to-many mappings (e.g., \u00df \u2192 SS).\n        \nCase_Ignorable: Characters skipped when determining case boundaries.\n    \n\n    \n    \n6. Libraries & Databases\n\n    \n6.1 The Gold Standard: ICU\n    \nMaintained by the Unicode Consortium and IBM, ICU is the definitive reference implementation.\n    \n\n        \nICU4C (C/C++) & ICU4J (Java): Deep property lookups via u_hasBinaryProperty() or u_getIntPropertyValue().\n        \nicu4x: Modern Rust-native re-implementation for embedded systems and WebAssembly.\n    \n\n    \n6.2 Language-Specific Ecosystems\n    \n\n        \n            LanguageLibraryDescription\n        \n        \n            Rustunicode-propertiesDirect access to General_Category, Script, Script_Extensions, Bidi_Class.\n            Rustunicode-segmentationGrapheme Cluster Break\n            Rustunicode-segmentationGrapheme Cluster Break and Word Break properties.\n            Rustucd-parse / ucd-trieParse raw UCD .txt files into O(1) lookup trie structures.\n            Pythonunicodedata (stdlib)Exposes category(), name(), bidirectional().\n            PythonPyICUBinds to full C++ ICU for rare properties like Script_Extensions.\n            Gogolang.org/x/text/unicodeParsed UCD tables (bidi, norm, rangetable, runes).\n            C/C++libicuucCore C library from ICU.\n            C/C++utf8cpp / simdutfSpecialized in UTF-8 decoding performance, validation, fast category conversions.\n        \n    \n\n    \n6.3 Raw Source Data: Unicode Character Database (UCD)\n    \nPublished at https://www.unicode.org/Public/UCD/latest/ucd/.\n    \n\n        \n            UCD Source FileProperties Included\n        \n        \n            UnicodeData.txtCode point value, Name, General Category, Canonical Combining Class, Bidi Class, Decomposition Mapping, Numeric Values, Simple Case Mappings.\n            Scripts.txtMaps code point ranges to primary Script property.\n            ScriptExtensions.txtMaps code points to Script_Extensions arrays.\n            PropList.txtBinary properties: White_Space, Alphabetic, Emoji, Default_Ignorable_Code_Point.\n            DerivedCoreProperties.txtAggregated properties: Grapheme_Base, Case_Ignorable.\n        \n    \n\n    \n    \n7. General Category Examples\n\n    \nSpecific code points for key General Category subcategories:\n\n    \nSk \u2014 Symbol, Modifier\n    \nNon-letter symbols that modify adjacent characters or act as standalone diacritics.\n    \n\n        \n            Code PointCharacterName\n        \n        \n            U+02C6\u02c6MODIFIER LETTER CIRCUMFLEX ACCENT\n            U+02C9\u02c9MODIFIER LETTER MACRON\n            U+1F3FB\ud83c\udffbEMOJI MODIFIER FITZPATRICK TYPE-1-2 (skin-tone modifier)\n            U+00A8\u00a8DIAERESIS\n        \n    \n\n    \nSo \u2014 Symbol, Other\n    \nDingbats, symbols, emojis, and visual glyphs not falling under Sm, Sc, or Sk.\n    \n\n        \n            Code PointCharacterName\n        \n        \n            U+1F600\ud83d\ude00GRINNING FACE\n            U+2602\u2602UMBRELLA\n            U+2665\u2665BLACK HEART SUIT\n            U+2622\u2622RADIOACTIVE SIGN\n        \n    \n\n    \nZl \u2014 Separator, Line\n    \nSpecifies explicit line breaks outside of control characters. This category consists of exactly one character.\n    \n\n        \n            Code PointCharacterName\n        \n        \n            U+2028\u2014LINE SEPARATOR (forces line break without new paragraph)\n        \n    \n\n    \nZp \u2014 Separator, Paragraph\n    \nSpecifies explicit paragraph breaks. Also exactly one character.\n    \n\n        \n            Code PointCharacterName\n        \n        \n            U+2029\u2014PARAGRAPH SEPARATOR\n        \n    \n\n    \nPo \u2014 Punctuation, Other\n    \nGeneral punctuation marks that are not dashes, brackets, connectors, or quotes.\n    \n\n        \n            Code PointCharacterName\n        \n        \n            U+0021!EXCLAMATION MARK\n            U+003F?QUESTION MARK\n            U+0023#NUMBER SIGN\n            U+0026&AMPERSAND\n            U+203D\u203dINTERROBANG\n        \n    \n\n    \nCs \u2014 Other, Surrogate\n    \nCode points reserved for the UTF-16 surrogate mechanism. Invalid in UTF-8 or UTF-32.\n    \n\n        \n            Code PointNameDescription\n        \n        \n            U+D800High SurrogateStart of range U+D800 \u2013 U+DBFF\n            U+DB7FHigh Private Use Surrogate\u2014\n            U+DC00Low SurrogateStart of range U+DC00 \u2013 U+DFFF\n            U+DFFFLow SurrogateEnd of range\n        \n    \n\n    \n    \n8. Unicode for Language/DSL Developers\n\n    \nWhen building a programming language or DSL compiler, supporting Unicode in the lexical analyzer goes beyond decoding UTF-8 bytes. Official standards from the Unicode Consortium handle identifiers, whitespace, security, and comments.\n\n    \n8.1 Identifiers: Use XID_Start and XID_Continue\n    \nDo not invent custom regex patterns like [a-zA-Z_\\u0080-\\uFFFF]. Implement Unicode Standard Annex #31 (UAX #31: Unicode Identifier and Pattern Syntax).\n\n    \n\n        Identifier := Start_Char (Continue_Char)*\n    \n\n    \n\n        \n            PropertyDescription\n        \n        \n            XID_StartLetters, ideographs, letter-like numbers across all world scripts. Excludes symbols and digits.\n            XID_ContinueSuperset of XID_Start plus combining marks, non-spacing diacritics, decimal numbers, connector punctuation.\n        \n    \n\n    \n\n        Why the X?\n        Always use XID_Start / XID_Continue rather than older ID_Start / ID_Continue. The X variants guarantee stability under Normalization Form C (NFC), ensuring an identifier won't alter its lexical meaning when normalized.\n    \n\n    \nCommon Language Profile: Most modern languages (Rust, Swift, C++23) tailor UAX #31 by adding ASCII _ (U+005F) to XID_Start.\n\n    \n8.2 Lexical Disjointness: Whitespace & Syntax Patterns\n    \nUAX #31 provides two specialized, mutually disjoint sets of code points:\n    \n\n        \nPattern_White_Space: Exact set of code points the lexer treats as syntax whitespace (spaces, tabs, LF, CR, U+2028/U+2029).\n        \nPattern_Syntax: Exact set of reserved syntax symbols across all scripts ({, }, +, =, punctuation, math operators).\n    \n    \nBecause XID_Continue, Pattern_White_Space, and Pattern_Syntax are guaranteed disjoint, a parser can cleanly distinguish syntax operators from variable names regardless of script.\n\n    \n8.3 Identifier Normalization & Comparison\n    \nDifferent code point sequences can look visually identical (pre-composed \u00e9 U+00E9 vs. decomposed e + \u25cc\u0301 U+0065 U+0301). The compiler must compare identifiers accurately:\n    \n\n        \nNormalize to NFC: Normalize all input identifiers to NFC before interning or placing them in the symbol table.\n        \nAvoid NFKC for Code: NFKC strips structural formatting (superscripts x\u00b2 \u2192 x2).\n        \nStrict NFC Equality: If identifier_A.to_NFC() == identifier_B.to_NFC(), they are the same variable.\n    \n\n    \n8.4 Security & Confusables (UTS #39)\n    \nArbitrary Unicode identifiers introduce severe security vectors like homograph attacks (e.g., defining adm\u0456n using Cyrillic \u0456 U+0456 alongside Latin admin).\n    \nImplement Unicode Technical Standard #39 (UTS #39: Unicode Security Mechanisms):\n    \n\n        \nMixed-Script Detection: Reject identifiers blending incompatible scripts in a single token (Latin + Cyrillic + Greek).\n        \nConfusable Checking: Compute the skeleton of identifiers using UTS #39 tables; warn or error if two declared identifiers' skeletons collide.\n    \n\n    \n8.5 Comments & BiDi Attacks (\"Trojan Source\")\n    \n\n        \ud83d\udee1\ufe0f CVE-2021-42574 \u2014 The Trojan Source Vulnerability\n        Attackers place invisible bidirectional formatting control characters (e.g., U+202E RIGHT-TO-LEFT OVERRIDE) inside string literals or comments. This alters the visual order of code in text editors compared to how the compiler's lexer consumes bytes linearly, hiding malicious code inside what appears to be a comment.\n    \n    \nCompiler Recommendations:\n    \n\n        \nOption A (Strict): Error or strip unescaped Bidirectional Control Characters (U+202A\u2013U+202E, U+2066\u2013U+2069) in comments and strings unless explicitly escaped.\n        \nOption B (Balanced): Require all BiDi control sequences inside comments to be properly balanced and terminated (using U+2069 POP DIRECTIONAL ISOLATE) before the comment boundary ends.\n    \n\n    \n    \n9. Math Literals, Operators & Keywords\n\n    \nWhen extending a language or DSL with Unicode math literals, custom operators, or non-ASCII keywords, the lexer faces strict grammar boundaries. UAX #31 and Unicode Technical Report #25 (Unicode Support for Mathematics) prevent lexical collisions and preserve AST determinism.\n\n    \n9.1 Symbol-Based Operators: The Pattern_Syntax Rule\n    \nFor Unicode operators (\u2260, \u2264, \u2208, \u2297, \u03bb), use explicit property classes rather than arbitrary ranges.\n    \nThe Gold Standard: Filter custom operator tokens using Pattern_Syntax = True or General Category subcategories Sm (Symbol, Math) and So (Symbol, Other).\n    \n\n        Why Pattern_Syntax Matters\n        Code points marked with Pattern_Syntax are permanently reserved by Unicode for language syntax. Pattern_Syntax and XID_Continue are strictly disjoint \u2014 a character can be a valid identifier constituent or a valid syntax symbol, but never both.\n    \n\n    \nOperator Canonicalization: Normalize all operators to NFC at the token level. The character \u2260 (U+2260) must not collide with or parse differently than its decomposed equivalent =\u0338 (U+003D U+0338).\n\n    \n\n        \n            Operator TypePrimary Unicode Properties / CategoriesExamples\n        \n        \n            Math OperatorsGeneral_Category = Sm+, \u2212 (U+2212), \u00d7 (U+00D7), \u00f7 (U+00F7), \u221a (U+221A), \u2208 (U+2208)\n            Arrows & RelationsBlock = Arrows, Supplemental_Arrows_A/B\u2192 (U+2192), \u21d2 (U+21D2), \u21a6 (U+21A6)\n            Punctuation OperatorsGeneral_Category = Po or Pd::, \u2026 (U+2026), \u2218 (U+2218)\n        \n    \n\n    \n9.2 Math Literals & Mathematical Alphanumeric Symbols\n\n    \nA. Multi-Digit Numbers & Numeric Properties\n    \n\n        \nGeneral_Category = Nd (Number, Decimal Digit): Includes decimal digits across scripts (Arabic-Indic \u0660-\u0669, Devanagari \u0966-\u096f).\n        \nAvoid No (Number, Other) in main number lexers: No includes vulgar fractions like \u00bd (U+00BD) and superscripts like \u00b2 (U+00B2). Lexing \u00bd as a decimal token crashes standard string-to-float parsers unless the grammar adds a fractional token type.\n    \n\n    \nB. Mathematical Alphanumeric Symbols (U+1D400 \u2013 U+1D7FF)\n    \nUnicode includes dedicated mathematical glyph variants in Plane 1, such as \ud835\udc00 (U+1D400 Mathematical Bold Capital A) or \ud835\udc65 (U+1D434 Mathematical Italic Small X).\n    \n\n        \u26a0\ufe0f NFKD Warning\n        Never run NFKD on math tokens. Compatibility normalization maps \ud835\udc65 (U+1D44E) directly to ASCII x (U+0078), destroying vector vs. scalar distinction in mathematical DSLs.\n    \n    \nIdentifier Inclusion: Under UAX #31, Mathematical Alphanumeric Symbols possess the XID_Start property. They are valid identifier characters by default, allowing mathematical DSLs to treat \ud835\udc65 and x as separate variables in the symbol table.\n\n    \n9.3 Non-ASCII Keywords (Localized DSLs)\n    \nFor non-ASCII keywords (e.g., funci\u00f3n in Spanish, \u51fd\u6570 in Chinese):\n    \n\n        \nLexing Order: Match input tokens against the keyword table after resolving identifier boundaries via XID_Start / XID_Continue.\n        \nCase Folding: For case-insensitive keywords, use Unicode Full Case Folding (Simple_Case_Folding fails for complex scripts) before checking the keyword hash map.\n        \nNormalization: Store keywords in the symbol table in NFC. Normalize source text to NFC before comparing identifier tokens against keyword sets.\n    \n\n    \nSummary Checklist for Parser Developers\n    \n\n        \n            Token TypeProperty / Category RuleNormalization\n        \n        \n            IdentifiersXID_Start + XID_Continue* (plus ASCII _)NFC\n            OperatorsPattern_Syntax = True or General_Category = SmNFC\n            Number LiteralsGeneral_Category = Nd for digits; isolate No into separate token classesNFC\n            WhitespacePattern_White_Space = True\u2014\n        \n    \n\n    \n    \n10. UAX #34 Relevance\n\n    \nUAX #34 (Unicode Named Character Sequences) is not directly relevant to parsing lexical tokens like identifiers, operators, or whitespace. However, it has niche relevance if your language provides a character escape mechanism by name.\n\n    \nWhat UAX #34 Actually Is\n    \nUAX #34 standardizes Named Character Sequences \u2014 pre-defined multi-code-point sequences assigned a single formal name in the Unicode database (NamedSequences.txt).\n    \nSome linguistic entities require multiple code points (base + combining marks) because they don't exist as a single pre-composed code point. UAX #34 defines official names for these exact combinations:\n    \n\n        \nLATIN SMALL LETTER I WITH MACRON AND GRAVE \u2192 <U+012B, U+0300>\n        \nKHMER CONSONANT SIGN COENG KA \u2192 <U+17D2, U+1780>\n    \n\n    \nIs It Relevant for Your Compiler / DSL Lexer?\n    \n\n        \n            ScenarioRelevant?Explanation\n        \n        \n            Core lexing & parsing\u274c NoUAX #34 does not define identifier boundaries or syntax properties. Use UAX #31.\n            Unicode properties\u274c NoNamed sequences do not carry individual General_Category or Script properties; those belong to underlying code points.\n            \\N{...} string/char escapes\u2705 YesIf your language supports escape sequences by character name (like Python, Rust, C++23).\n        \n    \n\n    \nNamed Sequence Escapes\n    \nIf a developer writes \\N{LATIN SMALL LETTER I WITH MACRON AND GRAVE}, a UAX #34-compliant compiler emits two code points (U+012B U+0300) as a single logical character literal.\n\n    \n\n        Key Takeaway\n        If you are writing core tokenization rules for variable names, syntax operators, or comments, ignore UAX #34 and focus on UAX #31. You only need UAX #34 if your lexer's string/char literal parser resolves Unicode character names (\\N{...}) to code point sequences.\n    \n\n    \n    \n11. Sources\n    \n\n        \nUAX #31: Unicode Identifier and Pattern Syntax\n        \nUAX #34: Unicode Named Character Sequences\n        \nUTS #39: Unicode Security Mechanisms\n        \nUTS #51: Unicode Emoji\n        \nUnicode Character Database (UCD)\n        \nICU \u2014 International Components for Unicode\n        \nThe Unicode Standard, Chapter 3: Conformance\n        \nPlane (Unicode) \u2014 Wikipedia\n        \nThe Unicode Character Set \u2014 NUMA\n        \nCVE-2021-42574 \u2014 Trojan Source\n        \nUTR #25: Unicode Support for Mathematics\n    \n\n    \n\n    \n\n        Advanced Unicode \u2014 Comprehensive Technical Reference \u00b7 Compiled from Unicode Consortium standards and industry best practices.\n    \n\n\n\n\n", "creation_timestamp": "2026-09-12T00:30:55.031416Z"}