Unicode and bidirectional source rules for the ASPL MVP

Archived MVP specification. For ASPL 0.7.0, see the current reference and native HTTP guide.

Researched on 2026-08-28 against Unicode 17.0, Unicode Technical Standard #55, and current language specifications.

Decision summary

ASPL should define .aspl files as UTF-8 and reject malformed byte sequences. The MVP should pin its Unicode data to version 17.0 instead of inheriting whatever tables happen to exist on the host system.

Identifiers should be case-sensitive and compared after NFC normalization. The lexer should use a documented profile of XID_Start and XID_Continue, add _, and restrict names to Arabic or Latin script. One identifier should not mix Arabic and Latin in the MVP, although a source file may freely contain Arabic and Latin tokens. Combining marks remain legal and significant. Arabic-Indic digits, non-ASCII connector punctuation, default-ignorable characters, and characters outside the Unicode General Security Profile should be rejected in identifiers.

The compiler should warn when distinct identifiers are confusable with each other or with an Arabic or English keyword. A confusable skeleton is a warning tool, never an identifier key.

Bidirectional source needs a display rule as well as a lexer rule. Store and parse tokens in logical order. Permit U+200E LEFT-TO-RIGHT MARK and U+200F RIGHT-TO-LEFT MARK only as semantically invisible controls at token boundaries. Do not let either character split one identifier into two. Preserve explicit directional controls inside strings and comments but warn on raw occurrences and render them visibly in diagnostics. Reject them in executable text outside those contexts.

The website editor should implement the atom-based source display from Unicode Technical Standard #55, with RTL and LTR atom order as a user preference. The compiler must still produce safe diagnostics for terminals that know nothing about ASPL tokens.

Standards baseline

Unicode Standard Annex #31 defines the default identifier grammar with XID_Start and XID_Continue. It recommends the XID properties over ID properties, especially where security matters, and allows a language to publish a narrower profile. It also requires a conforming profile to name its Unicode version and rules. Unicode Standard Annex #31.

Unicode Technical Standard #55 applies those rules to source code. For a new general-purpose language that does not require forward compatibility with unknown future characters, it recommends default identifiers, equivalent normalized identifiers, Unicode whitespace handling, token-aware bidirectional display, and confusability diagnostics. Unicode Technical Standard #55.

Unicode Technical Standard #39 supplies the General Security Profile, script detection, and confusable skeleton algorithms. It warns that confusability data can change as fonts and Unicode data improve. Unicode Technical Standard #39.

These documents solve different problems:

UAX #31 decides whether a sequence can be an identifier.
UAX #15 defines canonical equivalence and normalization.
UAX #9 determines visual bidirectional ordering.
UTS #39 finds suspicious characters, script mixtures, and lookalikes.
UTS #55 connects those pieces to programming language source and diagnostics.

Source encoding and line boundaries

Define an .aspl file as a sequence of Unicode scalar values encoded in UTF-8. Reject overlong forms, surrogate encodings, truncated sequences, and every other malformed UTF-8 sequence. RFC 3629 requires decoders to protect against invalid sequences because permissive decoding can have security consequences. RFC 3629.

Accept an optional UTF-8 BOM only at byte offset zero and remove it before lexing. Reject U+FEFF elsewhere in executable source. String literals may still represent that value through an escape if the language defines one. Unicode treats a UTF-8 BOM as an optional encoding signature, not a byte-order instruction, and notes that U+FEFF should not normally occur in the middle of a file. Unicode UTF and BOM FAQ.

Recognize all seven UAX #31 line terminators as line breaks: LF, VT, FF, CR, NEL, LINE SEPARATOR, and PARAGRAPH SEPARATOR. Treat CRLF as one line break. UTS #55 recommends this because an editor may display these characters as hard line breaks even when a compiler does not. A language that supports fewer must reject every unsupported line terminator, including one hidden inside a line comment. UAX #31 whitespace rules and UTS #55 language guidance.

This rule does not require rewriting the source buffer. The lexer can map all accepted line terminators to the same logical newline token while retaining original byte offsets for diagnostics.

Identifier profile

Use this profile, based on Unicode 17.0:

Start with _ or a code point having XID_Start=Yes.
Continue with code points having XID_Continue=Yes.
Permit only characters whose Script_Extensions are compatible with Arabic or Latin. Common and Inherited characters remain eligible only when the rest of this profile admits them.
Require the identifier's resolved strong script to be Arabic or Latin. Do not allow both strong scripts in one identifier in the MVP.
Of all Nd characters, permit only ASCII 0 through 9, and only after the first character.
Of all Pc characters, permit only _.
Reject every Default_Ignorable_Code_Point, including variation selectors, ZWJ, and ZWNJ.
Reject characters with Identifier_Status=Restricted in the Unicode 17.0 General Security Profile.

UAX #31 defines the XID classes and explicitly supports language-specific profiles. UTS #39's security profile marks unassigned, private-use, deprecated, default-ignorable, non-XID, obsolete, technical, and uncommon characters as Restricted. UAX #31 default identifiers and UTS #39 General Security Profile.

The single-script rule is an intentional MVP restriction, not a Unicode requirement. UTS #55 notes that mixed-script names such as HTTPЗапрос can be legitimate and defines a chunk-based diagnostic for languages that allow them. ASPL does not need that complexity to support Arabic names, English names, or Arabic code that calls an English C API. A later language version can allow visibly separated Arabic and Latin chunks without changing the rest of the profile. UTS #55 mixed-script guidance.

Keep combining marks that survive the profile. Do not strip Arabic vowel marks or treat them as insignificant. UTS #55 warns language designers not to remove the Mn and Mc categories wholesale because many writing systems require them. ASPL's NFC comparison handles canonically equivalent sequences while leaving a genuinely different marked name distinct. UTS #55 identifier guidance.

Rejecting ZWJ and ZWNJ is also a deliberate MVP boundary. UTS #39 defines contextual rules that make them usable in Persian and several other orthographies, and UTS #55 recommends a modified security profile for languages that need those names. Supporting them correctly requires joining-type and script-context checks. ASPL can add that profile later if Persian or another Arabic-script language becomes part of the stated audience. UTS #39 joining controls.

Do not add mathematical styled letters or compatibility notation to the MVP profile. UTS #55 recommends that broader profile for scientific languages, but ASPL's chosen MVP has ASCII operators and only a tiny procedural core. Adding it now would conflict with the project's stated syntax boundary and expand confusable cases.

Normalization and identifier equality

Normalize each complete identifier to NFC before keyword lookup, symbol-table insertion, equality checks, bytecode serialization, and linking decisions. Preserve the original spelling and byte span for diagnostics. Two source spellings with the same NFC form name the same declaration.

UAX #15 says normalization gives equivalent text a unique binary representation. UTS #55 recommends NFC for case-sensitive languages and NFKC plus full case folding for case-insensitive languages. It also says implementations should treat normalized identifiers as equivalent instead of requiring users to type pre-normalized text, because input methods may emit decomposed sequences. Unicode Standard Annex #15 and UTS #55 normalization guidance.

This makes NFC the right choice for ASPL if Latin identifiers remain case-sensitive. Arabic has no upper and lower case, so the rule imposes no case burden on Arabic names. If ASPL later chooses case-insensitive Latin identifiers, it must revisit the identity rule and use the UTS #55 NFKC case-folding model.

Normalize only identifiers and keyword candidates. Do not silently normalize string or character literal contents, comments, file paths, C symbol names, or user data. UTS #55 makes the same distinction for string literals. A source formatter may offer an explicit NFC rewrite, but compilation must retain string data exactly. UTS #55 normalization guidance.

The profile also rejects Arabic presentation forms and compatibility ligatures as identifier spellings when they fail XID or the security profile. Their ordinary Arabic letter sequences remain available, and strings can contain the presentation characters when exact data matters.

Rust provides a useful primary-language precedent for this combination. Its reference uses XID_Start, XID_Continue, an added underscore, NFC equality, and a ban on ZWJ and ZWNJ. Rust identifier specification. Python shows the main alternative by applying NFKC to names, but that compatibility-based choice is not the current UTS #55 recommendation for a case-sensitive language. Python lexical specification.

Bidirectional controls and lexical meaning

Source is stored in logical order. The sequence of tokens in the file is the sequence parsed by the compiler, regardless of where an editor paints each glyph. The Unicode Bidirectional Algorithm controls display, not lexical order. Unicode Standard Annex #9.

UAX #31 includes LRM and RLM in Pattern_White_Space as ignorable format controls. It recommends allowing them between lexical elements so source can display correctly as plain text. Their insertion must not change program meaning, and neither mark should count as the visible whitespace required to separate two word-like tokens. UAX #31 bidirectional whitespace guidance.

Implement that rule narrowly:

Ignore LRM and RLM at the start or end of a lexical line, beside real whitespace, or where removing the mark still leaves a token boundary.
If removing a mark would join two identifier pieces, report an illegal control inside an identifier. ret<LRM>urn must not become a keyword, and foo<LRM>bar must not become two names.
Do not accept LRM or RLM as the only separator where the grammar requires visible whitespace.
Reject ALM and explicit embedding, override, or isolate controls in executable text outside a string or comment.
Preserve those controls inside strings and comments. Warn on every raw occurrence and suggest an escape when the value is intentional.

A full UTS #55 implementation may accept more directional formatting controls and diagnose only controls whose effect crosses lexical atoms. The narrower MVP rule is easier to audit while still admitting the two implicit marks used to repair mixed Arabic and ASCII code. UTS #55 directional-formatting guidance.

The warning set should at least name these explicit controls: LRE, RLE, LRO, RLO, PDF, LRI, RLI, FSI, and PDI. Rust's compiler uses deny-by-default lints for these code points in comments and literals, which is evidence that plain syntax highlighting is not enough protection. Rust text-direction lint documentation.

Display contract for the website and tools

UTS #55 divides each source line into lexical atoms. Tokens are atom boundaries. String and comment delimiters are separate from their contents, ASCII numeric literals stay intact, and each identifier is one atom. The editor then orders atoms consistently from left to right or right to left and applies the Unicode Bidirectional Algorithm inside each atom. It must not infer the whole file's atom order from the first strong character. UTS #55 basic ordering.

The ASPL website should implement that model. Default to RTL atom order when the Arabic interface is active and LTR when the English interface is active, but let the user switch without changing the file. In HTML, UTS #55 recommends a separate span for each atom and a matching dir attribute, with numeric atoms forced to dir="ltr" and comment content using dir="auto".

This display rule belongs in the playground acceptance criteria. A plain dir="rtl" text area does not preserve token order reliably when Arabic identifiers, ASCII operators, punctuation, and Western digits share a line.

The compiler and formatter should also offer a safe plain-text representation. Never print a raw directional control into a terminal diagnostic. Show its escape, code point, and Unicode name. When echoing a source line, either apply the UTS #55 conversion-to-plain-text algorithm to a temporary diagnostic rendering or replace controls with visible labels. Do not rewrite the user's file without an explicit formatting command. UTS #55 conversion to plain text.

Confusable identifiers

After normalization and legality checks, compute the UTS #39 confusable form for every declared identifier and every Arabic and English keyword. Warn when two distinct visible names in the same file share a confusable form, or when an identifier is confusable with a keyword. Do not warn when the names are already equal under NFC because they are one identifier, not two lookalikes.

UTS #39 defines direction-aware bidiSkeleton operations and says the ordinary skeleton is the LTR form. It also warns that skeleton values are internal comparison values, are not suitable for display, and can change between Unicode versions. UTS #39 confusable detection.

Because ASPL's editor supports both atom orders, the compiler should warn when either the LTR or RTL skeleton collides. The website can refine the warning for the user's selected display direction. Store no skeleton in bytecode or public symbol identity. Recompute warnings from the pinned tables.

Confusable diagnostics should remain warnings. UTS #39's mappings evolve, and visual similarity depends on fonts. Identifier legality, NFC equality, and the single-script rule provide the stable hard boundary. Rust follows a similar split by accepting non-ASCII identifiers while warning about UTS #39 confusable pairs and uncommon code points. Rust confusable identifier lint.

Diagnostic contract

Unicode diagnostics need more than a caret under a terminal line. Each identifier or control-character diagnostic should include:

the source span as UTF-8 byte offsets
the logical line number and scalar-value column
the original spelling and NFC spelling when they differ
each suspicious code point in U+XXXX NAME form
an escaped rendering of invisible and bidi-control characters
the other declaration or keyword involved in a confusable collision
a concrete replacement when ASPL requires ASCII digits or punctuation

For a normalized duplicate, point to both declarations and say that their NFC forms are identical. For a script violation, name the detected scripts. For a rejected Arabic-Indic digit, suggest the corresponding ASCII digit. Arabic and English diagnostics should carry the same fields and error identity so documentation and tests can compare them.

The website can highlight the original byte span using its source-to-display mapping. Terminal output should favor an unambiguous escaped excerpt over a visually polished but reordered line.

Versioning and generated data

Vendor or generate the needed Unicode 17.0 tables as part of the compiler source. Do not use the process locale or the host C library's character classification. The compiler needs versioned data for XID properties, normalization, Script_Extensions, identifier status, bidi classes, joining types, character names used in diagnostics, and confusable mappings.

Record Unicode 17.0 in compiler version output and in the language specification. UAX #31's XID stability rules mean existing default identifiers remain identifiers as Unicode grows. ASPL's narrower profile and UTS #39 data still need explicit upgrade tests. A Unicode table update is a language change, not an invisible dependency refresh.

Conformance tests implied by the research

The lexer and diagnostic suite should cover at least these cases:

Malformed UTF-8, overlong sequences, surrogates, and a misplaced BOM fail before tokenization.
Canonically equivalent NFC and decomposed identifiers resolve to one symbol.
Strings preserve decomposed text instead of normalizing it.
Arabic and Latin identifiers work in one file, while one mixed Arabic-Latin identifier fails.
Common Arabic combining marks remain legal and significant.
Arabic-Indic digits, compatibility presentation forms, tatweel when Restricted, ZWJ, and ZWNJ fail in identifiers.
LRM and RLM work at legal token boundaries but cannot split an identifier or replace required visible space.
Raw explicit directional controls outside strings and comments fail. Raw controls inside them produce named warnings and safe excerpts.
Distinct identifiers confusable in LTR or RTL display warn, including collisions with either keyword vocabulary.
Every accepted line terminator ends a line, including inside a line comment.
The website renders the same token sequence correctly in LTR and RTL atom order.

These tests turn Arabic support into a language contract. Without them, Unicode behavior would depend on the terminal, editor, C library, or whichever Unicode data happened to be installed.