Choose the compiler pipeline and internal contracts

Archived MVP specification. For ASPL 0.5.0, see the current reference and native HTTP guide.

Type: grilling Status: resolved Blocked by: 06, 08, 09, 10 Parent: Find the ASPL MVP specification

Question

Which compiler stages and representations should connect UTF-8 source, bilingual tokens, parsing, static checking, bytecode generation, execution, and ELF packaging, and what contracts between those stages make the C implementation testable without prematurely fixing its file layout?

Answer

The compiler core is a buffer-oriented sequence of explicit, owned representations:

source bytes
  -> SourceText
  -> LexedSource
  -> ParsedProgram
  -> CheckedProgram
  -> BytecodeModule
  -> canonical byte buffer
  -> bytecode validator
  -> VmProgram

A successful compile returns CompiledBytecode, which owns the canonical byte buffer, the immutable VmProgram decoded from those exact bytes, and an optional in-memory SourceMap. Execution and packaging then split:

VmProgram -> HostPreparedProgram -> VmExecution -> VmOutcome

CompiledBytecode + validated VM template -> completed ELF byte buffer

The stage names describe logical contracts. They do not prescribe C filenames, headers, struct layouts, allocation strategy inside a representation, or a public library ABI.

Common stage contract

Every major representation is opaque outside its owning module and immutable once construction succeeds. It owns all of its allocated storage and has an explicit destructor. A stage may pass stable integer IDs, half-open source spans, and read-only views across a boundary, but it may not expose pointers into growable storage or require another module to understand its struct layout.

The original SourceText remains available when diagnostics or SourceMap entries need rendered source. Later representations store numeric spans rather than borrowed pointers into the source buffer. CheckedProgram owns every normalized name, literal value, type, and resolved identity needed for lowering, so lowering does not depend on ParsedProgram or LexedSource storage.

Every stage has:

• one construction operation with explicit input, limits, allocator, and cancellation control;
• read-only query or visitor operations for consumers and tests;
• an invariant checker that accepts only complete results; and
• a destructor that is safe after any successful construction.

No compiler or VM stage reads a file, writes a file, prints a diagnostic, exits a process, consults the process locale, or starts another program. The Linux CLI owns filesystem and terminal behavior. The browser worker passes memory buffers. Linux foreign-library operations live behind the native host adapter.

All mutable state belongs to one compilation, preparation, or execution context. Separate contexts may run concurrently. A single context need not accept concurrent calls. Generated Unicode, parser, opcode, adapter, and other release tables may be immutable process globals.

Allocations use an explicit checked allocator interface. Every allocation failure unwinds owned state and returns a non-source failure. This interface must support deterministic fault injection in tests.

SourceText

SourceText owns the exact input bytes and a caller-supplied display label. It performs the already-settled strict UTF-8 and optional leading BOM checks before tokenization, then records every accepted line boundary. LexedSource applies the pinned Unicode 17.0 lexical tables. A malformed UTF-8 sequence produces no LexedSource.

All source spans are half-open byte ranges measured in the original input. When a BOM is present, the first following token starts at byte offset three. Lexing otherwise ignores the BOM, while the lossless view may identify its span. SourceText builds the mapping needed to report logical one-based lines and scalar-value columns for all accepted line terminators, with CRLF counted as one break.

The display label and source contents never enter bytecode or an ELF payload.

LexedSource

LexedSource is produced by one scan and exposes two ordered views:

• significant tokens consumed by the parser; and
• a lossless lexical-atom and trivia index used by diagnostics, formatting, and token-aware display.

Every entry carries its original SourceText span. Arabic and English keyword spellings normalize to the same significant token kind, while the span preserves the spelling the user wrote. Legal LRM and RLM boundary controls remain visible in the lossless view and disappear from the significant-token view.

Identifier tokens own their NFC-normalized UTF-8 name. String tokens own their decoded, exact UTF-8 value after escape processing, without normalizing the contents. Numeric tokens retain a lexically valid source spelling and classification. The checker, not the lexer or parser, decides whether an integer magnitude or floating value is semantically valid. This preserves the special minimum-i64 case.

Comments, whitespace, delimiter atoms, and other trivia do not enter ParsedProgram. Lexical recovery may produce several diagnostics and continue at a valid scalar or token boundary, but a LexedSource with lexical errors cannot produce a CheckedProgram.

ParsedProgram

ParsedProgram is a compact, span-rich syntax tree. It represents the source grammar after keyword-alias normalization. It is not a lossless concrete syntax tree and contains no comments or trivia.

Parsing does not resolve names, assign types, evaluate literal ranges, choose bytecode operations, allocate registers, or lower structured control flow. It retains the written forms of conditionals, loops, Boolean operators, inequality, calls, bindings, assignments, and returns so later source diagnostics can name the construct the user wrote.

The parser recovers at grammar-defined boundaries to report several deterministic syntax errors. Static checking starts only when lexing and parsing complete without errors.

CheckedProgram

The checker first installs the release-fixed builtin catalog, then collects all ASPL function signatures and foreign declarations before checking every body. Declaration collection is an internal checker pass, not another cross-module representation. It permits source-order-independent calls and mutual recursion while still checking bodies in source order.

CheckedProgram is a self-contained, immutable, typed structured IR:

• every function, foreign declaration, builtin, parameter, local, and call target has a stable ID;
• every expression has exactly one settled type;
• every name use resolves to one declaration or binding;
• mutability, unsafe markers, arity, conversions, returns, structural reachability, and all semantic-core rules have been checked;
• integer and floating constants have their final bit patterns, and strings have their final bytes; and
• source spans remain attached to constructs that may need compile-time diagnostics or instruction mapping.

It preserves structured if and while forms. It contains no bytecode registers, branch targets, table indices, host capability decisions, or serialized layouts.

The checker checks unused functions and foreign declarations. It returns no CheckedProgram when any source error exists, but warnings may accompany a successful result.

Results and diagnostics

Each stage returns an owning StageResult of its logical result type. It contains structured diagnostic records and either a complete result or no result. A diagnostic record carries a stable reason key, severity, primary and secondary spans, and typed arguments. It never carries final Arabic or English prose. The diagnostics decision owns the catalog, wording, presentation, and error-recovery cap.

Zero errors permits a result. Warnings do not block it. Cancellation, configured resource exhaustion, host failure, allocation failure, and violated internal invariants are distinct result statuses rather than source diagnostics.

Malformed UTF-8 stops before lexing. Lexing and parsing recover enough to report independent errors. Checking starts only after lexical and syntax success, may report independent semantic errors, and produces no CheckedProgram after an error. Lowering and serialization have no ordinary source-error outcomes because CheckedProgram already proves their source invariants.

Language, host, and execution policies

The checker consumes one immutable, release-fixed language catalog. It is not a plugin registry. That catalog contains the اطبع and print aliases for the single (string) -> unit output builtin fixed by Choose the MVP builtin environment. CheckedProgram uses one builtin target identity for both spellings. The lowerer emits the matching fixed bytecode operation rather than using an extension hook.

Three separate limit inputs govern different work:

• CompileLimits covers source bytes, lexical items, syntax nodes, nesting, diagnostic count, and compiler working memory.
• HostPolicy covers bytecode size and table counts plus host capabilities such as Linux foreign imports.
• ExecutionLimits covers instructions, frames, slots, VM memory, output, and host-controlled time.

The native CLI and browser host contracts choose their numeric limits. Each layer checks counts and size arithmetic in a widened type before allocating or narrowing.

Static checking is host-neutral. A host-capability gate runs after checking and before lowering. The browser policy rejects any foreign declaration that would create a bytecode import. The bytecode loader also rejects an externally supplied program with imports when the selected host forbids them. This second check prevents raw bytecode from bypassing the compiler gate.

HostPolicy changes acceptance and limits only. It does not change instruction selection, table ordering, serialization, or language semantics. Source accepted by both native and browser policies therefore produces identical bytecode.

An optional cancellation probe passes through compilation, host preparation, and execution. The lexer polls while scanning, the parser and checker poll at bounded node boundaries, lowering polls at bounded construct boundaries, and the VM polls before instruction dispatch. Cancellation returns its own status, destroys partial state, and never publishes partial bytecode or ELF output. A browser may still terminate its worker when foreign or faulty native code prevents cooperative cancellation.

Lowering and BytecodeModule

The MVP has no optimizer stage. Lowering emits every checked ASPL function and foreign declaration, including declarations unreachable from main. It does not fold constants, eliminate dead code, inline functions, reorder evaluation, or remove unused declarations.

Lowering performs only the transformations required by the fixed bytecode contract:

• resolve function, import, and builtin target IDs;
• assign typed registers;
• stage call arguments into contiguous ranges;
• lower structured control flow and compound source operators;
• build canonical constant and string tables; and
• resolve all symbolic labels to function-relative instruction indices.

Mutable builders, symbolic labels, and fixups are private to the lowerer. Completed BytecodeModule is the final logical bytecode. It stores host-native integer values and arrays rather than packed file bytes. All registers, table references, instruction operands, and branch targets already have their serialized meanings.

The BytecodeModule invariant checker enforces every compiler-side fact that does not require parsing bytes. The serializer makes no semantic decision. It writes the fixed section order, little-endian scalar fields, alignment padding, exact sizes, and CRC required by the bytecode contract.

Canonical identity and table order

Lowering uses these deterministic rules:

• ASPL functions appear in source definition order. The bytecode header points to the source position of the entry function.
• Foreign imports use the settled library-bytes, symbol-bytes, then adapter-ID order.
• i64 and f64 constants are deduplicated by exact 64-bit pattern and sorted by that pattern interpreted as an unsigned integer.
• String values are deduplicated by exact UTF-8 bytes and sorted lexicographically by unsigned byte value.
• The string-bytes section concatenates each unique string once in string-reference order. The compiler emits no overlapping string references.
• Ordinary function names use their NFC identifier bytes. The entry function name always uses the canonical Arabic bytes for ابدأ_الرئيسية, whether the source used ابدأ_الرئيسية or start_main.
• Register-type and instruction ranges follow function-table order as required by the bytecode format.

These rules make repeated compilation with the same release, source bytes, and accepted policy byte-identical. Keyword-alias and resolved builtin-alias changes, whitespace, and comments also leave bytecode unchanged when they do not change identifiers, literals, declarations, or declaration order.

Register assignment

The MVP does not reuse temporary registers.

Parameters occupy the first registers in declaration order, as the bytecode contract already requires. Local bindings follow in lexical declaration order across the function. Remaining registers are assigned monotonically during deterministic left-to-right lowering.

Every expression that yields a value to its surrounding checked construct receives a typed result register. A discarded call emits the discard form and needs no result register. A name reference copies its binding or parameter value into its expression register. Initializers and assignments evaluate completely before copying into the local register, which preserves the settled evaluation and mutation rules.

Before lowering an ASPL or foreign call's arguments, the lowerer reserves one contiguous staging range with the exact parameter types. It evaluates arguments left to right and copies each result into its staging register before moving to the next argument. The call result uses a separate register, even though the bytecode format permits overlap. Zero-argument calls use the required zero first-argument operand.

The output builtin evaluates its one string argument normally, then emits WRITE_STRING with that result register. It does not reserve a staging range, create a frame, add an import, or allocate a result register.

This strategy spends registers and COPY instructions to avoid a liveness allocator. Register limits remain explicit HostPolicy checks, so exceeding a selected host's cap is a compile-time resource failure rather than a different allocation strategy.

Canonical control flow

Each structured source form has one lowering pattern.

An if without else emits its condition, JUMP_IF_FALSE to the first instruction after the true block, then the true block. An if with else targets the first else instruction and emits a JUMP over the else block only when the true block can fall through. The lowerer omits an end label when both branches terminate.

A while target begins at the first instruction that evaluates its condition. JUMP_IF_FALSE targets the first instruction after the loop. The body emits a back edge only when it can fall through. The checker does not treat a loop as proof of a return, so a valid non-unit function always has a reachable instruction after the false edge.

Boolean and lowers as:

evaluate left
load false into result
jump to end when left is false
evaluate right
copy right into result
end

Boolean or lowers as:

evaluate left
load true into result
jump to right when left is false
jump to end
right:
evaluate right
copy right into result
end

Inequality emits the matching typed equality instruction followed by NOT_BOOL.

The lowerer tracks whether every block can fall through. It emits no instruction after an unconditional return on the same path, no redundant jump from a terminated block, and no instruction that the bytecode validator would reject as unreachable. A reachable natural end of a unit function receives RETURN_UNIT. Empty source blocks share the next real instruction as their branch target rather than creating a NOP, which bytecode version 1 does not have.

Validation and CompiledBytecode

The compiler serializes BytecodeModule, then passes the complete byte buffer through the canonical bytecode validator. The validator remains the only constructor of VmProgram.

If the validator rejects fresh compiler output, compilation returns an internal compiler failure. If the same validator rejects external bytes or bytes loaded from an ELF payload, the caller reports a load failure. The distinction comes from provenance, not from separate validators.

On success, CompiledBytecode owns:

• the exact canonical byte buffer;
• the immutable VmProgram decoded and validated from those bytes; and
• an optional SourceMap for the current source compilation.

SourceMap keys instruction entries by final function index and function-relative instruction index. It also keys foreign declaration spans by final import index so native preparation failures can identify the responsible source declaration. It never enters serialized bytecode or ELF. Direct aspl run execution and the browser playground may use it. Standalone bytecode and generated native programs report bytecode function and instruction indices or import details without source spans.

Mapping uses the smallest responsible source construct:

• literal loads use the literal span;
• arithmetic, comparison, conversion, and inequality instructions use the operator or conversion span;
• ASPL calls, foreign calls, their staging copies, and WRITE_STRING use the call span;
• conditional branches use the condition span;
• loop back edges use the while span;
• short-circuit support instructions use the Boolean operator span;
• explicit return instructions use the return statement span; and
• an implicit RETURN_UNIT uses the function's closing-brace span.

Every foreign import entry uses its complete foreign declaration span.

The diagnostics decision may choose how much of this information to show, but the pipeline preserves it.

Execution and packaging boundaries

VmProgram is immutable, host-neutral, and contains no open libraries or mutable registers. Host preparation applies HostPolicy and produces HostPreparedProgram. The native implementation eagerly opens and resolves the complete sorted import table according to the dynamic-C decision. The browser implementation has no foreign loader and accepts only a program permitted by its empty foreign capability.

HostPreparedProgram owns native library handles and resolved function pointers for as long as any execution may use them. Each VmExecution owns its call frames, register slots, counters, ExecutionLimits, cancellation control, and host callbacks. Its synchronous output callback borrows string bytes only for one call and reports success, an output limit, or a host failure as fixed by Choose the MVP builtin environment. One execution produces one structured VmOutcome. The CLI and browser adapters translate that outcome into their own process status, displayed diagnostics, output, or UI state.

ELF packaging consumes CompiledBytecode and the validated release template. It never receives ParsedProgram, CheckedProgram, HostPreparedProgram, or VmExecution. It copies the canonical byte buffer unchanged into the native envelope, validates the completed ELF and its embedded bytecode, then returns an owned byte buffer to the CLI publication layer. It performs no file publication itself.

Native startup validates the native envelope, passes the embedded bytes through the same bytecode validator, prepares the native host, and executes. It has no SourceMap because the ELF format deliberately contains no source mapping.

Test contract

Tests observe opaque results through read-only queries and deterministic contributor-only structural dumps. Dumps are snapshot aids, not public formats or compatibility promises. Tests must not depend on allocation addresses, struct padding, hash-table iteration, C filenames, or header organization.

The pipeline suite must cover at least:

• exact SourceText byte, line, scalar-column, BOM, and line-terminator mapping;
• Arabic, English, and mixed keyword aliases producing the same significant token kinds while the lossless view preserves each spelling;
• اطبع and print resolving to one builtin target and producing byte-identical WRITE_STRING bytecode;
• lossless lexical coverage of comments, trivia, controls, and original spans;
• parser recovery without invoking the checker after lexical or syntax errors;
• checker collection of order-independent declarations and rejection of every semantic acceptance case already listed;
• absence of a CheckedProgram or later artifact after any source error;
• complete emission of unused functions and foreign declarations;
• the fixed table orders, register assignment, and control-flow templates;
• byte-identical output across repeated runs and across keyword-only alias changes;
• BytecodeModule invariant failures, serializer boundary values, and validator round trips;
• rejection of deliberately corrupted serialized bytes through the external-input path;
• browser capability rejection without changing bytecode for programs accepted by both hosts;
• source maps that cover every emitted instruction, including WRITE_STRING, and every import while never entering serialized output;
• cancellation and every configured limit at each relevant stage;
• allocator failure injected at every allocation site with complete cleanup;
• repeated execution from immutable validated state with fresh VmExecution state; and
• ELF packaging that preserves the canonical bytecode bytes and revalidates the completed artifact.

Scope boundaries

The MVP exposes no stable C embedding ABI. Its Linux CLI, native template, tests, and WebAssembly wrapper are version-matched internal consumers. The browser behavior decision owns the small JavaScript buffer interface, and the CLI decision owns the supported command interface.

The pipeline has no plugin stages, configurable passes, incremental compilation, persistent cache, serialized syntax tree, serialized checked IR, or on-disk intermediate files. Each invocation processes one in-memory source file with one release-fixed language catalog.

Exact diagnostic keys and wording, numeric limits, CLI flags and publication behavior, browser UI and stop behavior, benchmark thresholds, and any fixed built-in entries remain with their existing decision tickets. Those choices plug into the contracts above without changing the stage boundaries.