Set the performance budget and benchmark protocol

Set the performance budget and benchmark protocol

Type: grilling Status: resolved Blocked by: 08, 10, 11, 13, 14 Parent: Find the ASPL MVP specification

Question

Which reference machine, fixture programs, cold and warm conditions, timing boundaries, output-size checks, browser measurements, and regression thresholds make the sub-100-ms small-program build goal reproducible and useful during implementation?

Answer

The native performance promise applies to the complete aspl build command. An optimized, uninstrumented release binary must build the fixed canonical small program at a warm-cache p95 below 100 milliseconds on the reference machine. The benchmark starts a fresh process for every sample and includes atomic ELF publication. Hosted CI detects relative regressions, but it does not decide the absolute promise. Browser performance has its own budgets.

These are release and regression gates, not execution limits exposed by the product. They do not add a native CLI timeout or change the browser's existing 2-second check and 3-second execution ceilings.

Reference machine

The physical reference machine is the current development host:

• Intel Core i7-6820HQ, four physical cores and eight hardware threads, with a maximum reported frequency of 3.6 GHz;
• 16 GiB of RAM;
• Micron 1100 SATA 512 GB SSD; and
• the Btrfs benchmark filesystem mounted with compress=zstd:3.

At the time of this decision, it runs Omarchy 4.0.1, Linux 7.1.9-arch1-2, glibc 2.44, and CPU microcode revision 0xf0. Those software versions describe the initial baseline, not a permanently frozen operating-system image.

Native benchmark runs use the installed host directly, without a container. The machine must be on AC power with the performance governor, Turbo enabled, and SMT enabled. The benchmark driver and its child inherit affinity to logical CPU 2. Its SMT sibling, logical CPU 6, must remain idle. No competing user workload or active swapping is allowed. A preflight records and checks these conditions. If they change during a batch, the harness rejects the whole batch.

The source and destination stay in one dedicated directory on the reference Btrfs filesystem. Browser benchmarks use the same power and workload rules, but do not pin the browser's multi-process workload to one logical CPU.

Fixture set

The repository must contain one fixed canonical native fixture. Its exact UTF-8 bytes are versioned and hashed. It is at most 2 KiB, uses canonical Arabic keyword aliases, follows the canonical formatting rules, and compiles without warnings. Several functions exercise all five storable types, immutable and mutable bindings, calls, integer and floating arithmetic, comparisons, if, while, and string values. It also declares and calls puts through the canonical Arabic سي_صحيح__مؤشر_محرف_ثابت foreign adapter name. Building the fixture must not open libc.so.6.

A fixed browser counterpart retains the ordinary program logic but removes the foreign declaration and call because the browser policy rejects foreign imports.

Secondary fixtures cover a minimal Arabic program, its equivalent English form, a deliberately mixed-vocabulary form, both foreign adapter profiles, and generated scaling cases. They diagnose which path regressed. They do not redefine the public small-program claim or its absolute gate.

Native timing boundary

The normative command is the optimized release candidate's aspl build with explicit fixture and output paths. The output exists before each measured invocation, so the command exercises replacement. The harness uses umask 022 and a fixed minimal environment. It invokes aspl directly without a shell.

The harness reads CLOCK_MONOTONIC_RAW immediately before process creation and again after waitpid reaps it. The interval therefore includes process creation and startup, argument processing, source opening and reading, the complete compiler and bytecode-validation pipeline, ELF construction and validation, temporary-file creation and writing, mode setting, close, atomic rename, and normal process exit.

Host preflight, cache preparation, untimed warm-ups, and post-run artifact inspection sit outside the interval. After every sample, the harness requires status 0, empty stdout and stderr, mode 0755, a structurally valid artifact, the expected byte count, and the expected content hash. Any failure rejects the complete batch rather than disappearing from the timing data.

The same harness reports end-to-end aspl check time for the canonical fixture as diagnostic evidence, with no product ceiling. Native run has no performance benchmark because program execution and foreign-library behavior would dominate it. A separately instrumented compiler build may report stage timings while investigating a regression, but those results cannot pass or fail the product gate.

Warm and cold protocols

The warm-cache protocol starts with 10 unmeasured complete builds. It then collects 200 measured builds. Every build starts a fresh process and overwrites the existing destination, while the executable, source, template, and filesystem metadata remain cache-warm.

The cold-cache protocol collects 20 samples. Before each sample, the harness completes sync and asks Linux to evict the page cache. Those administrative preparation steps are outside the timed interval, and aspl still runs as the ordinary benchmark user. Each timed sample starts a fresh process and overwrites an existing destination. Cold results report median and p95 only; the MVP assigns them no pass or fail ceiling.

The harness retains every raw duration in nanoseconds. It sorts the values and uses the nearest-rank definition, with rank ceil(p * N), for percentiles. Published summaries show minimum, median, p95, p99, and maximum in milliseconds to three decimal places, plus the raw samples.

The release gate passes only when the 200-sample warm-cache p95 is strictly below 100 milliseconds. A p95 at or above 90 milliseconds produces a headroom warning. The harness never removes an individual slow result. A host-condition violation or functional failure invalidates the whole batch and requires a new one.

Development regression checks

Hosted CI builds the merge base and candidate outside the measured interval, warms each with 10 unmeasured invocations, then alternates 50 measured samples of each in one job. The job fails if either condition is true:

• candidate median time is both more than 10 percent and more than 2 milliseconds above the merge base; or
• candidate p95 time is both more than 15 percent and more than 5 milliseconds above the merge base.

An intentional slowdown requires a reviewed baseline update that explains the trade-off. Hosted CI does not apply the absolute 100-millisecond gate because runner performance varies. The physical reference machine remains the authority for release acceptance.

Native artifact checks

Every benchmark result records the source, canonical bytecode, embedded VM template, and final ELF byte counts and SHA-256 hashes. The packaging invariants are:

payload segment bytes = 24 + canonical bytecode bytes
final ELF bytes       = embedded template bytes + canonical bytecode bytes

The second equation does not add another 24 bytes because the embedded template already contains the initial 24-byte payload envelope.

Repeated builds from the same release binary, source bytes, and options must be byte-identical. Across revisions, the performance check fails when either threshold is crossed:

• canonical bytecode grows by both more than 5 percent and more than 256 bytes; or
• final ELF output grows by both more than 5 percent and more than 4 KiB.

An intentional increase requires a reviewed baseline update that identifies its source. The MVP sets no arbitrary absolute ELF-size ceiling before implementation establishes a measured baseline. The exact packaging equations and byte-for-byte determinism always remain mandatory.

Browser protocol and budgets

The browser harness runs the fixed foreign-free fixture in exact documented Firefox and Chromium releases on the reference machine. It uses headed browsers, clean profiles, normal hardware acceleration, and production assets served over loopback HTTP. This isolates application loading from wide-area network variance. Mobile devices remain in layout and functional conformance testing, but outside the performance promise.

Each warm browser measurement has 10 unmeasured warm-ups followed by 100 samples. Cold navigation has 20 samples with the HTTP cache disabled. The harness uses monotonic browser timing and the same nearest-rank percentile rule as the native harness. Both browsers must meet every p95 ceiling:

The check round trip starts when the main thread dispatches the job to an already loaded worker and stops when it accepts the matching result for the current source revision. Initial module loading and the edit debounce are not part of this interval. The Run measurement includes the required complete recheck but stops before VM execution. Execution has its own interval.

The browser result also records raw and compressed JavaScript and WebAssembly sizes and peak WebAssembly linear memory for the small fixture. A change requires reviewed baseline approval when an asset grows by both more than 5 percent and more than 16 KiB, or peak linear memory grows by both more than 10 percent and more than 1 MiB.

Evidence and baseline changes

Every retained result identifies the physical hardware, firmware and microcode, Omarchy build, kernel, glibc, CPU governor, Turbo and SMT state, filesystem and mount options, power state, browser versions where applicable, compiler commit, release-binary hash, fixture hashes, benchmark parameters, raw samples, summaries, and artifact measurements.

An operating-system, kernel, glibc, or browser update creates a new environment identity. The project reruns both the last accepted ASPL commit and the candidate in that new environment, establishes a new versioned baseline, and retains the old records. It never compares unmatched environments as if they were one time series. Replacing the physical reference machine requires a new explicit performance decision.