From fc34bd9b2498587b7d78bb0b3da881c15cb794e3 Mon Sep 17 00:00:00 2001 From: Oleksandr Kozachuk <201152+ok2@users.noreply.github.com> Date: Sun, 9 Aug 2026 09:20:10 +0200 Subject: [PATCH 1/3] perf(core): typed calling convention for words with a known stack effect Such a word now compiles to a fast entry (i32 x p) -> (i32 x q) carrying its stack items as WASM values, plus the usual ( -- ) wrapper that keeps the table slot, so EXECUTE / interpreter / host words / CATCH see the unchanged memory ABI. Fib(25) 1035 -> 366 us, 4.3x slower than sf64 -> 1.2x; the default guards-on config 1740 -> 361 us. WS-006. --- CHANGELOG.md | 47 ++ README.md | 25 +- crates/cli/src/main.rs | 4 +- crates/core/src/codegen.rs | 932 ++++++++++++++++++++++++++++---- crates/core/src/config.rs | 7 + crates/core/src/consolidate.rs | 18 +- crates/core/src/export.rs | 1 + crates/core/src/outer.rs | 75 +++ crates/core/tests/compliance.rs | 29 + 9 files changed, 1024 insertions(+), 114 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 85847ec..1926b5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,53 @@ All notable changes to WAFER are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +- **A typed calling convention for words with a known stack effect.** Such a + word now compiles to two entry points: a fast one whose signature is + `(i32 x p) -> (i32 x q)`, carrying its stack items as WASM values, and the + usual `( -- )` wrapper that moves those items on and off the memory data + stack. The wrapper keeps the function-table slot, so `EXECUTE`, the outer + interpreter, host words and `CATCH` see exactly the ABI they saw before; + only direct calls inside a module take the fast entry. + + This is what the SwiftForth gap was made of. sf64 keeps TOS in `RBX` and + the stack pointer in `RBP`, and both survive a `CALL` untouched, so its + `FIB` is 16 instructions and ~7 memory touches per node. WAFER kept the + whole stack in linear memory and flushed its cached `$dsp` to an imported + global before every call: ~36 memory touches per node. The stack simulator + that already promoted loop and `IF` bodies into WASM locals refused any + body containing a call or an `EXIT` -- exactly the words where the + convention cost the most. It now handles both. + + Fibonacci(25) goes from 1035 to 366 µs, 4.3x slower than `sf64` to 1.2x. + Loop-heavy benchmarks are unchanged (they were already promoted, and + already beat `sf64`). Words that keep the memory convention: anything + using `SP@`, `DEPTH`, `EXECUTE`, `>R`/`R>`, floats or locals; anything + calling a word that is itself untyped, which in the JIT path means every + call except `RECURSE`; mutually recursive words; and words whose effect is + not static -- branches that disagree on depth, `EXIT` at the wrong depth, + a non-neutral loop body, or a recursion that grows the stack per level. + + `CONSOLIDATE` extends this across words, since it puts them all in one + module: the effects are solved to a fixpoint from the leaves outward, and + 105 of 187 words in a booted dictionary end up typed. + + Stack guards get cheap as a side effect -- they hang off the memory-stack + push/pop choke points, and a typed word barely has any. The default + guards-on configuration that the REPL and the web build use went from 1631 + to 365 µs on the same benchmark. + + `WAFER_TYPED_CALLS=0` falls back to the memory-stack convention. + +### Fixed + +- The Forth 2012 Core suite now also runs against consolidated code + (`compliance_core_after_consolidate`). `CONSOLIDATE` had no correctness + test at all before -- only benchmarks. + ## [0.2.6] - 2026-08-07 ### Fixed diff --git a/README.md b/README.md index 0574cf7..4ba3c59 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,7 @@ An optimizing Forth 2012 compiler targeting WebAssembly. WAFER JIT-compiles each - **Faster than gforth** on all benchmarks in release mode (2-10x faster) - **JIT compilation** — each `:` definition compiles to its own WASM module - **Self-recursive direct calls** — RECURSE compiles to native `call` instead of `call_indirect` +- **Typed calling convention** — a word with a statically known stack effect passes its stack items as WASM values, so a call keeps them in registers instead of round-tripping through memory - **Consolidation mode** — recompile all words into a single optimized WASM module - **Interactive REPL** with line editing (rustyline) - **Browser REPL** — runs entirely in the browser via wasm-pack + js-sys @@ -79,23 +80,31 @@ git submodule update --init ## Performance -WAFER beats gforth (the GNU Forth reference implementation) on all benchmarks in release mode: +WAFER beats gforth (the GNU Forth reference implementation) on all benchmarks in release mode, and is within +reach of SwiftForth `sf64`, which compiles to native code: ``` -Benchmark WAFER CONSOL gforth WAFER/gf -Fibonacci(25) 1629 1535 3422 0.45x -Factorial(12)x10K 340 339 638 0.53x -GCD-bench(500) 18 15 30 0.50x -NestedLoops(50) 84 73 720 0.10x -Collatz(2K) 1212 1202 3914 0.31x +Benchmark WAFER CONSOL gforth sf64 WAFER/gf WAFER/sf +Fibonacci(25) 378 359 3238 296 0.11x 1.21x +Factorial(12)x10K 335 320 633 183 0.51x 1.75x +GCD-bench(500) 14 15 29 31 0.48x 0.45x +NestedLoops(50) 72 69 698 207 0.10x 0.33x +Collatz(2K) 994 997 3940 668 0.25x 1.49x ``` Times in microseconds. WAFER/gf < 1.0 means WAFER is faster. CONSOL = after `CONSOLIDATE`. +A word whose stack effect is statically known gets a **typed entry point**: its stack items travel in and out +as WASM values instead of through the memory data stack, so cranelift keeps them in registers across a call +the way a native Forth keeps TOS in one. The word also keeps a `( -- )` wrapper, which is what the function +table, `EXECUTE` and the outer interpreter reach, so nothing about the memory ABI changes from the outside. +Call-heavy code is what this pays for -- Fibonacci went from 4.3x slower than `sf64` to 1.2x. Set +`WAFER_TYPED_CALLS=0` to fall back to the memory-stack convention. + ## Testing ```bash -# All tests (~570 currently passing) +# All tests (~620 currently passing) cargo test --workspace # Forth 2012 compliance suite diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index 96d6840..1436650 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -263,7 +263,8 @@ fn cmd_run(file: &str) -> anyhow::Result<()> { } /// `WaferConfig` for CLI-created VMs. `WAFER_STACK_GUARDS=0|1` overrides -/// the per-command default (REPL/file execution on, build off). +/// the per-command default (REPL/file execution on, build off); +/// `WAFER_TYPED_CALLS=0` falls back to the memory-stack calling convention. fn vm_config(default_guards: bool) -> wafer_core::config::WaferConfig { let mut cfg = wafer_core::config::WaferConfig::all(); cfg.codegen.stack_guards = match std::env::var("WAFER_STACK_GUARDS").ok().as_deref() { @@ -271,6 +272,7 @@ fn vm_config(default_guards: bool) -> wafer_core::config::WaferConfig { Some(_) => true, None => default_guards, }; + cfg.codegen.typed_calls = std::env::var("WAFER_TYPED_CALLS").ok().as_deref() != Some("0"); cfg } diff --git a/crates/core/src/codegen.rs b/crates/core/src/codegen.rs index f9dd20d..e2ef3fa 100644 --- a/crates/core/src/codegen.rs +++ b/crates/core/src/codegen.rs @@ -8,6 +8,7 @@ use std::borrow::Cow; use std::collections::HashMap; +use std::rc::Rc; use wasm_encoder::{ BlockType, CodeSection, ConstExpr, CustomSection, DataCountSection, DataSection, @@ -46,11 +47,15 @@ const TABLE: u32 = 0; // Type indices in the type section. const TYPE_VOID: u32 = 0; // () -> () const TYPE_I32: u32 = 1; // (i32) -> () +const TYPE_TYPED: u32 = 2; // (i32 x p) -> (i32 x q), single-word modules only // The `emit` callback is the first (and only) imported function, so index 0. -// The compiled word is the first (and only) defined function, so index 1. +// The compiled word is the first defined function, so index 1. const EMIT_FUNC: u32 = 0; const WORD_FUNC: u32 = 1; +/// Fast entry of a typed word in a single-word module, right after the +/// `() -> ()` wrapper that keeps the table slot. +const TYPED_FAST_FUNC: u32 = 2; // --------------------------------------------------------------------------- // DSP caching: local 0 holds a cached copy of the $dsp global. @@ -100,6 +105,23 @@ pub struct CodegenConfig { /// Table index of the `_STACK_FAULT_` host word; `Some` enables /// stack under/overflow guards in the emitted code. pub stack_guards: Option, + /// Give words with a statically known stack effect a typed entry point + /// that passes stack items as WASM values instead of through memory. + pub typed_calls: bool, +} + +/// A word compiled with the typed calling convention: its stack items travel +/// as WASM values instead of through the memory data stack, so cranelift can +/// keep them in registers across a call the way a native Forth keeps TOS in +/// one. Reachable only by direct `call` inside the same module. +#[derive(Debug, Clone, Copy)] +struct TypedFn { + /// WASM function index of the fast entry. + fn_index: u32, + /// Cells taken from the caller. + params: u32, + /// Cells handed back. + results: u32, } /// Result of compiling a word to WASM. @@ -1244,13 +1266,28 @@ fn is_promotable(ops: &[IrOp]) -> bool { if ops.is_empty() { return false; } - is_promotable_body(ops) + is_promotable_body(ops, PromoteMode::Memory) +} + +/// Which promoted code path a body is being checked for. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum PromoteMode { + /// Classic promotion: the word keeps the memory-stack ABI, so it may not + /// call anything (the callee would see a stale stack) and may not `EXIT` + /// (the promoted locals have to be written back first). + Memory, + /// Typed entry: the word takes and returns its stack items as WASM + /// values, so calls to other typed words and `EXIT` are both fine. + Typed, } /// Recursive check for promotable ops. -fn is_promotable_body(ops: &[IrOp]) -> bool { +fn is_promotable_body(ops: &[IrOp], mode: PromoteMode) -> bool { + let typed = mode == PromoteMode::Typed; for op in ops { match op { + IrOp::Call(_) | IrOp::TailCall(_) if typed => {} + IrOp::Exit if typed => {} IrOp::Call(_) | IrOp::TailCall(_) | IrOp::Execute | IrOp::SpFetch | IrOp::RpFetch => { return false; } @@ -1288,31 +1325,46 @@ fn is_promotable_body(ops: &[IrOp]) -> bool { then_body, else_body, } => { - let Some(eb) = else_body else { - return false; - }; - if !is_promotable_body(then_body) || !is_promotable_body(eb) { + // In typed mode a missing ELSE is fine and the depth + // agreement is checked by `analyze_stack`, which knows the + // call effects and which branches EXIT. + if !is_promotable_body(then_body, mode) { return false; } - // Both branches must have the same net stack effect - let (_, then_net) = compute_stack_needs(then_body); - let (_, else_net) = compute_stack_needs(eb); - if then_net != else_net { - return false; + match else_body { + Some(eb) => { + if !is_promotable_body(eb, mode) { + return false; + } + if !typed { + // Both branches must have the same net stack effect + let (_, then_net) = compute_stack_needs(then_body); + let (_, else_net) = compute_stack_needs(eb); + if then_net != else_net { + return false; + } + } + } + None if typed => {} + None => return false, } } // DO/LOOP: promotable if body is promotable and stack-neutral IrOp::DoLoop { body, is_plus_loop } => { - if !is_promotable_body(body) { + if !is_promotable_body(body, mode) { return false; } - if body.iter().any(|op| matches!(op, IrOp::Exit)) { + // An EXIT out of a DO loop would have to unwind the loop + // locals; neither path emits that, so leave those words alone. + if body_has_exit(body) { return false; } - let (_, body_net) = compute_stack_needs(body); - let expected = if *is_plus_loop { 1 } else { 0 }; - if body_net != expected { - return false; + if !typed { + let (_, body_net) = compute_stack_needs(body); + let expected = if *is_plus_loop { 1 } else { 0 }; + if body_net != expected { + return false; + } } } // BEGIN loops, BeginDoubleWhileRepeat, flat forward blocks: not promoted @@ -1330,6 +1382,166 @@ fn is_promotable_body(ops: &[IrOp]) -> bool { true } +/// Does `ops` contain an `EXIT` at any nesting depth? +fn body_has_exit(ops: &[IrOp]) -> bool { + ops.iter().any(|op| match op { + IrOp::Exit => true, + IrOp::If { + then_body, + else_body, + } => body_has_exit(then_body) || else_body.as_deref().is_some_and(body_has_exit), + IrOp::DoLoop { body, .. } | IrOp::BeginUntil { body } | IrOp::BeginAgain { body } => { + body_has_exit(body) + } + IrOp::BeginWhileRepeat { test, body } => body_has_exit(test) || body_has_exit(body), + IrOp::BeginDoubleWhileRepeat { + outer_test, + inner_test, + body, + after_repeat, + else_body, + } => { + body_has_exit(outer_test) + || body_has_exit(inner_test) + || body_has_exit(body) + || body_has_exit(after_repeat) + || else_body.as_deref().is_some_and(body_has_exit) + } + _ => false, + }) +} + +/// Every word `ops` calls, at any nesting depth. +fn callees_of(ops: &[IrOp], out: &mut Vec) { + for op in ops { + match op { + IrOp::Call(id) | IrOp::TailCall(id) => out.push(*id), + IrOp::If { + then_body, + else_body, + } => { + callees_of(then_body, out); + if let Some(eb) = else_body { + callees_of(eb, out); + } + } + IrOp::DoLoop { body, .. } | IrOp::BeginUntil { body } | IrOp::BeginAgain { body } => { + callees_of(body, out); + } + IrOp::BeginWhileRepeat { test, body } => { + callees_of(test, out); + callees_of(body, out); + } + IrOp::BeginDoubleWhileRepeat { + outer_test, + inner_test, + body, + after_repeat, + else_body, + } => { + callees_of(outer_test, out); + callees_of(inner_test, out); + callees_of(body, out); + callees_of(after_repeat, out); + if let Some(eb) = else_body { + callees_of(eb, out); + } + } + _ => {} + } + } +} + +/// Most cells a typed entry may take as WASM parameters. Beyond this the +/// signature stops fitting in argument registers and the memory stack is the +/// better deal. +const MAX_TYPED_PARAMS: u32 = 8; + +/// Most cells a typed entry may return. WASM multi-value returns past the +/// first go through a caller-provided return area, i.e. memory again, so +/// keep this tight. +const MAX_TYPED_RESULTS: u32 = 4; + +/// How many iterations to give the self-recursion fixpoint before giving up. +const TYPED_EFFECT_ROUNDS: u32 = 6; + +/// Solve a word's stack effect, if it has one. +/// +/// `known` supplies the effect of every word this one calls; `self_id` names +/// the word itself, whose effect is what we are solving for. A self-call +/// makes the equation circular (`d = k + m*d` for m self-calls), so the +/// analysis is run to a fixpoint: guess, re-derive, stop when the guess +/// reproduces itself. `: FIB ... RECURSE ... RECURSE ... ;` settles on +/// `(1, 1)` in two rounds; a word with no fixed effect (`: F 1 RECURSE ;`) +/// grows without settling and is rejected. +fn typed_effect( + body: &[IrOp], + self_id: Option, + known: &HashMap, +) -> Option { + if body.is_empty() || !is_promotable_body(body, PromoteMode::Typed) { + return None; + } + // Every call target must already have an effect, or be this word. + let mut targets = Vec::new(); + callees_of(body, &mut targets); + if targets + .iter() + .any(|id| Some(*id) != self_id && !known.contains_key(id)) + { + return None; + } + + let mut guess: CallEffect = (0, 0); + for _ in 0..TYPED_EFFECT_ROUNDS { + let lookup = |id: WordId| { + if Some(id) == self_id { + guess + } else { + known.get(&id).copied().unwrap_or((0, 0)) + } + }; + let (preload, net, consistent) = analyze_stack(body, &lookup); + if !consistent { + return None; + } + let produced = i32::try_from(preload).ok()? + net; + let effect = (preload, u32::try_from(produced).ok()?); + if effect == guess { + return (effect.0 <= MAX_TYPED_PARAMS && effect.1 <= MAX_TYPED_RESULTS) + .then_some(effect); + } + guess = effect; + } + None +} + +/// Solve the stack effect of every word in a consolidated module that has +/// one. +/// +/// A word can be typed once all the words it calls are, so this grows the +/// set until it stops growing: leaves first, then their callers. Mutually +/// recursive words never enter it -- neither can be settled before the +/// other -- and keep the memory-stack convention. +fn typed_effects(words: &[(WordId, Vec)]) -> HashMap { + let mut known: HashMap = HashMap::new(); + loop { + let mut changed = false; + for (id, body) in words { + if known.contains_key(id) { + continue; + } + if let Some(effect) = typed_effect(body, Some(*id), &known) { + known.insert(*id, effect); + changed = true; + } + } + if !changed { + return known; + } + } +} + /// Compute the net stack depth change for a single IR operation. fn stack_delta(op: &IrOp) -> i32 { match op { @@ -1392,26 +1604,73 @@ fn stack_delta(op: &IrOp) -> i32 { /// them (e.g., `Dup` reads the top). We must track the minimum stack position /// that any op reads from, not just the net depth after consumption. fn compute_stack_needs(ops: &[IrOp]) -> (u32, i32) { - let mut depth: i32 = 0; - let mut min_accessed: i32 = 0; - compute_stack_needs_rec(ops, &mut depth, &mut min_accessed); - let preload = if min_accessed < 0 { - (-min_accessed) as u32 + let (preload, net, _) = analyze_stack(ops, &|_| (0, 0)); + (preload, net) +} + +/// Run the stack-needs analysis with a stack effect for each called word. +/// +/// Returns `(preload, net, consistent)`. `consistent` is false when the body +/// has no static stack effect at all -- branches that disagree on depth, an +/// `EXIT` at a depth the fall-through path does not reach, a loop body that +/// is not stack-neutral. The classic memory-stack path ignores it (its +/// bodies contain no calls and no `EXIT`, so it is always true there); the +/// typed path refuses to compile a word without it. +fn analyze_stack(ops: &[IrOp], calls: &CallEffects<'_>) -> (u32, i32, bool) { + let mut st = Needs { + depth: 0, + min_accessed: 0, + diverged: false, + consistent: true, + exit_depth: None, + calls, + }; + compute_stack_needs_rec(ops, &mut st); + if let Some(d) = st.exit_depth + && !st.diverged + && d != st.depth + { + st.consistent = false; + } + let preload = if st.min_accessed < 0 { + (-st.min_accessed) as u32 } else { 0 }; - (preload, depth) + (preload, st.depth, st.consistent) +} + +/// Stack effect of a called word: (cells consumed, cells produced). +type CallEffect = (u32, u32); + +/// Look up the stack effect of a call target. Only ever consulted for words +/// [`is_promotable_body`] has already accepted in [`PromoteMode::Typed`], so +/// an unknown target cannot reach it. +type CallEffects<'a> = dyn Fn(WordId) -> CallEffect + 'a; + +/// Abstract-interpretation state for the stack-effect analysis. +struct Needs<'a> { + depth: i32, + min_accessed: i32, + /// Set once control cannot fall out of the body being walked (`EXIT`). + diverged: bool, + /// Cleared when the body has no static stack effect. + consistent: bool, + /// Depth the first `EXIT` left; every other exit must agree. + exit_depth: Option, + calls: &'a CallEffects<'a>, } /// Recursive stack-needs analysis that descends into control flow bodies. -fn compute_stack_needs_rec(ops: &[IrOp], depth: &mut i32, min_accessed: &mut i32) { +fn compute_stack_needs_rec(ops: &[IrOp], st: &mut Needs<'_>) { for op in ops { + let depth = st.depth; // First: compute the deepest position this op reads from. let reads_from = match op { - IrOp::Dup => *depth - 1, - IrOp::Over | IrOp::TwoDup => *depth - 2, - IrOp::Swap | IrOp::Nip | IrOp::Tuck => *depth - 2, - IrOp::Rot => *depth - 3, + IrOp::Dup => depth - 1, + IrOp::Over | IrOp::TwoDup => depth - 2, + IrOp::Swap | IrOp::Nip | IrOp::Tuck => depth - 2, + IrOp::Rot => depth - 3, IrOp::Add | IrOp::Sub | IrOp::Mul @@ -1429,7 +1688,7 @@ fn compute_stack_needs_rec(ops: &[IrOp], depth: &mut i32, min_accessed: &mut i32 | IrOp::DivMod | IrOp::Store | IrOp::CStore - | IrOp::PlusStore => *depth - 2, + | IrOp::PlusStore => depth - 2, IrOp::Drop | IrOp::Negate | IrOp::Abs @@ -1437,15 +1696,17 @@ fn compute_stack_needs_rec(ops: &[IrOp], depth: &mut i32, min_accessed: &mut i32 | IrOp::ZeroEq | IrOp::ZeroLt | IrOp::Fetch - | IrOp::CFetch => *depth - 1, - IrOp::TwoDrop => *depth - 2, - IrOp::FetchFloat | IrOp::StoreFloat | IrOp::StoF => *depth - 1, + | IrOp::CFetch => depth - 1, + IrOp::TwoDrop => depth - 2, + IrOp::FetchFloat | IrOp::StoreFloat | IrOp::StoF => depth - 1, // Control flow reads are handled by recursion below - IrOp::If { .. } => *depth - 1, // consumes condition - IrOp::DoLoop { .. } => *depth - 2, // consumes limit + index - _ => *depth, + IrOp::If { .. } => depth - 1, // consumes condition + IrOp::DoLoop { .. } => depth - 2, // consumes limit + index + // A call reads as deep as its own arguments go. + IrOp::Call(id) | IrOp::TailCall(id) => depth - (st.calls)(*id).0 as i32, + _ => depth, }; - *min_accessed = (*min_accessed).min(reads_from); + st.min_accessed = st.min_accessed.min(reads_from); // Then: update depth. For control flow, recurse instead of using stack_delta. match op { @@ -1453,48 +1714,67 @@ fn compute_stack_needs_rec(ops: &[IrOp], depth: &mut i32, min_accessed: &mut i32 then_body, else_body, } => { - *depth -= 1; // consume condition - let saved = *depth; - compute_stack_needs_rec(then_body, depth, min_accessed); + st.depth -= 1; // consume condition + let saved = st.depth; + + let outer_diverged = st.diverged; + st.diverged = false; + compute_stack_needs_rec(then_body, st); + let then_depth = st.depth; + let then_diverged = st.diverged; + + st.depth = saved; + st.diverged = false; if let Some(eb) = else_body { - let then_depth = *depth; - *depth = saved; - compute_stack_needs_rec(eb, depth, min_accessed); - // Use the then-branch depth (both should match for well-formed code) - *depth = then_depth; + compute_stack_needs_rec(eb, st); } + let else_depth = st.depth; + let else_diverged = st.diverged; + + // A branch that always EXITs never reaches the join, so it + // does not have to agree on depth with the one that does. + st.depth = if then_diverged { + else_depth + } else { + then_depth + }; + if !then_diverged && !else_diverged && then_depth != else_depth { + st.consistent = false; + } + st.diverged = outer_diverged || (then_diverged && else_diverged); } IrOp::DoLoop { body, is_plus_loop } => { - *depth -= 2; // consume limit + index - // Loop body is stack-neutral (net 0, or +1 for +LOOP step) - // We still recurse to track min_accessed inside the body. - let saved = *depth; - compute_stack_needs_rec(body, depth, min_accessed); - // Restore: body effect is consumed by loop control - *depth = saved; - if *is_plus_loop { - // +LOOP body pushes 1 step value, consumed by loop control + st.depth -= 2; // consume limit + index + // Loop body is stack-neutral (net 0, or +1 for +LOOP step: + // the step value is consumed by the loop control). + let saved = st.depth; + compute_stack_needs_rec(body, st); + let expected = saved + i32::from(*is_plus_loop); + if st.depth != expected { + st.consistent = false; } + // Restore: body effect is consumed by loop control + st.depth = saved; } IrOp::BeginUntil { body } => { - let saved = *depth; - compute_stack_needs_rec(body, depth, min_accessed); + let saved = st.depth; + compute_stack_needs_rec(body, st); // Body produces flag, consumed by UNTIL: net 0 for the whole construct - *depth = saved; + st.depth = saved; } IrOp::BeginAgain { body } => { - let saved = *depth; - compute_stack_needs_rec(body, depth, min_accessed); - *depth = saved; + let saved = st.depth; + compute_stack_needs_rec(body, st); + st.depth = saved; } IrOp::BeginWhileRepeat { test, body } => { - let saved = *depth; - compute_stack_needs_rec(test, depth, min_accessed); + let saved = st.depth; + compute_stack_needs_rec(test, st); // WHILE consumes flag - *depth -= 1; - compute_stack_needs_rec(body, depth, min_accessed); + st.depth -= 1; + compute_stack_needs_rec(body, st); // Whole construct is stack-neutral - *depth = saved; + st.depth = saved; } IrOp::BeginDoubleWhileRepeat { outer_test, @@ -1503,21 +1783,35 @@ fn compute_stack_needs_rec(ops: &[IrOp], depth: &mut i32, min_accessed: &mut i32 after_repeat, else_body, } => { - let saved = *depth; - compute_stack_needs_rec(outer_test, depth, min_accessed); - *depth -= 1; - compute_stack_needs_rec(inner_test, depth, min_accessed); - *depth -= 1; - compute_stack_needs_rec(body, depth, min_accessed); - compute_stack_needs_rec(after_repeat, depth, min_accessed); + let saved = st.depth; + compute_stack_needs_rec(outer_test, st); + st.depth -= 1; + compute_stack_needs_rec(inner_test, st); + st.depth -= 1; + compute_stack_needs_rec(body, st); + compute_stack_needs_rec(after_repeat, st); if let Some(eb) = else_body { - compute_stack_needs_rec(eb, depth, min_accessed); + compute_stack_needs_rec(eb, st); } - *depth = saved; + st.depth = saved; + } + IrOp::Call(id) | IrOp::TailCall(id) => { + let (consumed, produced) = (st.calls)(*id); + st.depth += produced as i32 - consumed as i32; + } + IrOp::Exit => { + // Every EXIT must leave the same depth, and the fall-through + // path has to reach it too (checked by `analyze_stack`). + match st.exit_depth { + None => st.exit_depth = Some(st.depth), + Some(d) if d != st.depth => st.consistent = false, + Some(_) => {} + } + st.diverged = true; } // All other ops: use stack_delta _ => { - *depth += stack_delta(op); + st.depth += stack_delta(op); } } } @@ -1594,6 +1888,8 @@ fn count_promoted_locals_body(ops: &[IrOp], count: &mut u32) { count_promoted_locals_body(eb, count); } } + // Typed path only: a call lands its results in fresh locals. + IrOp::Call(_) | IrOp::TailCall(_) => *count += MAX_TYPED_RESULTS, IrOp::Dup | IrOp::Over | IrOp::Tuck | IrOp::TwoDup => { // These reuse existing locals via the simulator, no extra needed } @@ -1611,6 +1907,21 @@ struct StackSim { next_local: u32, /// Stack of (`index_local`, `limit_local`) for nested DO/LOOP in promoted path. loop_index_stack: Vec<(u32, u32)>, + /// Set when emitting the fast entry of a typed word. `None` is the + /// classic memory-stack promotion, where calls and `EXIT` cannot occur. + typed: Option, + /// True once the code emitted so far cannot fall through (an `EXIT` ran). + /// The join after an `IF` uses it to take the surviving branch's state. + diverged: bool, +} + +/// What the typed emitter needs beyond the simulator itself. +#[derive(Clone)] +struct TypedCtx { + /// Cells this function returns, i.e. what an `EXIT` has to leave. + results: u32, + /// Fast entries reachable by direct call from this module. + callees: Rc>, } impl StackSim { @@ -1619,6 +1930,39 @@ impl StackSim { stack: Vec::new(), next_local: first_local, loop_index_stack: Vec::new(), + typed: None, + diverged: false, + } + } + + /// Simulator for a typed fast entry: params occupy locals `0..params`, + /// so fresh locals start above them. + fn new_typed(params: u32, results: u32, callees: &Rc>) -> Self { + let mut sim = Self::new(params); + sim.stack = (0..params).collect(); + sim.typed = Some(TypedCtx { + results, + callees: Rc::clone(callees), + }); + sim + } + + /// Emit the function's results and return. Used by `EXIT` and at the end + /// of a typed body. + fn emit_typed_return(&self, f: &mut Function, explicit_return: bool) { + let results = self.typed.as_ref().map_or(0, |t| t.results) as usize; + let Some(base) = self.stack.len().checked_sub(results) else { + // Too few values to return means every path here already + // returned, so the validator treats this position as + // unreachable and any terminator satisfies it. + f.instruction(&Instruction::Unreachable); + return; + }; + for &local in &self.stack[base..] { + f.instruction(&Instruction::LocalGet(local)); + } + if explicit_return { + f.instruction(&Instruction::Return); } } @@ -1941,34 +2285,47 @@ fn emit_promoted_op(f: &mut Function, op: &IrOp, sim: &mut StackSim) { let saved_stack = sim.stack.clone(); let saved_next = sim.next_local; + let outer_diverged = sim.diverged; + sim.diverged = false; emit_promoted_body(f, then_body, sim); let then_stack = sim.stack.clone(); let then_next = sim.next_local; + let then_diverged = sim.diverged; // Restore to branch-point state for else sim.stack = saved_stack; sim.next_local = saved_next; + sim.diverged = false; f.instruction(&Instruction::Else); if let Some(eb) = else_body { emit_promoted_body(f, eb, sim); } + let else_diverged = sim.diverged; - // Copy else results into then's locals at the join point. - // Both branches should have the same stack depth for well-formed Forth. - let else_stack = &sim.stack; - let min_len = then_stack.len().min(else_stack.len()); - for i in 0..min_len { - if then_stack[i] != else_stack[i] { - f.instruction(&Instruction::LocalGet(else_stack[i])); - f.instruction(&Instruction::LocalSet(then_stack[i])); + // A branch that returned never reaches the join, so its locals do + // not have to be reconciled -- the survivor's state is the join + // state. When both fall through, copy the else results into the + // then branch's locals (both have the same depth by construction). + if then_diverged { + // join state is the else state, already in sim.stack + } else { + if !else_diverged { + let else_stack = &sim.stack; + let min_len = then_stack.len().min(else_stack.len()); + for i in 0..min_len { + if then_stack[i] != else_stack[i] { + f.instruction(&Instruction::LocalGet(else_stack[i])); + f.instruction(&Instruction::LocalSet(then_stack[i])); + } + } } + sim.stack = then_stack; } - - sim.stack = then_stack; sim.next_local = sim.next_local.max(then_next); + sim.diverged = outer_diverged || (then_diverged && else_diverged); f.instruction(&Instruction::End); } @@ -2146,6 +2503,12 @@ fn emit_promoted_op(f: &mut Function, op: &IrOp, sim: &mut StackSim) { sim.push(result); } + IrOp::Exit if sim.typed.is_some() => { + // Typed entry: hand the results back as WASM values. + sim.emit_typed_return(f, true); + sim.diverged = true; + } + IrOp::Exit => { // Write remaining promoted locals back to memory stack, then return emit_promoted_epilogue(f, sim); @@ -2153,6 +2516,39 @@ fn emit_promoted_op(f: &mut Function, op: &IrOp, sim: &mut StackSim) { f.instruction(&Instruction::Return); } + // A call between typed words: arguments go in WASM parameters and + // results come back as WASM results, so nothing touches memory. + // `TailCall` is only ever generated in tail position, so emitting it + // as a plain call and falling through to the return is equivalent. + IrOp::Call(id) | IrOp::TailCall(id) if sim.typed.is_some() => { + // Both invariants are established by `typed_effect`, which + // refuses a body with an unknown callee and verifies the depth + // at every point -- same contract as `StackSim::pop`. + let callee = sim + .typed + .as_ref() + .and_then(|t| t.callees.get(id).copied()) + .expect("typed call to an untyped word"); + let base = sim + .stack + .len() + .checked_sub(callee.params as usize) + .expect("promoted stack underflow at a typed call"); + for &local in &sim.stack[base..] { + f.instruction(&Instruction::LocalGet(local)); + } + sim.stack.truncate(base); + f.instruction(&Instruction::Call(callee.fn_index)); + // Results arrive on the operand stack with the topmost last. + let results: Vec = (0..callee.results).map(|_| sim.alloc()).collect(); + for &local in results.iter().rev() { + f.instruction(&Instruction::LocalSet(local)); + } + for local in results { + sim.push(local); + } + } + // Unhandled ops in promoted path — shouldn't reach here if is_promotable is correct _ => {} } @@ -2165,6 +2561,58 @@ fn emit_promoted_body(f: &mut Function, ops: &[IrOp], sim: &mut StackSim) { } } +/// Build the fast entry of a typed word: stack items in, stack items out, +/// the memory data stack never touched. +fn emit_typed_fast( + body: &[IrOp], + effect: CallEffect, + callees: &Rc>, +) -> Function { + let (params, results) = effect; + // Params are locals 0..params; everything the simulator allocates on top + // of that has to be declared. + let extra = count_promoted_locals(body, 0) + results; + let mut f = Function::new(vec![(extra, ValType::I32)]); + let mut sim = StackSim::new_typed(params, results, callees); + emit_promoted_body(&mut f, body, &mut sim); + sim.emit_typed_return(&mut f, false); + f.instruction(&Instruction::End); + f +} + +/// Build the `() -> ()` wrapper that lets a typed word be reached the normal +/// way -- from the table, `EXECUTE`, the outer interpreter. It moves the +/// arguments off the memory data stack into the typed call and the results +/// back, which is also where the stack guards for the word live. +fn emit_typed_wrapper(effect: CallEffect, fast_index: u32) -> Function { + let (params, results) = effect; + let mut f = Function::new(vec![(1 + params + results, ValType::I32)]); + f.instruction(&Instruction::GlobalGet(DSP)) + .instruction(&Instruction::LocalSet(CACHED_DSP_LOCAL)); + + let mut sim = StackSim::new(SCRATCH_BASE); + emit_promoted_prologue(&mut f, params, &mut sim); + for &local in &sim.stack { + f.instruction(&Instruction::LocalGet(local)); + } + sim.stack.clear(); + f.instruction(&Instruction::Call(fast_index)); + + let out: Vec = (0..results).map(|_| sim.alloc()).collect(); + for &local in out.iter().rev() { + f.instruction(&Instruction::LocalSet(local)); + } + for local in out { + sim.push(local); + } + emit_promoted_epilogue(&mut f, &mut sim); + + f.instruction(&Instruction::LocalGet(CACHED_DSP_LOCAL)) + .instruction(&Instruction::GlobalSet(DSP)); + f.instruction(&Instruction::End); + f +} + /// At the end of a loop iteration in promoted code, copy modified values /// back into the loop-top locals so the next iteration reads correct values. fn emit_promoted_loop_fixup(f: &mut Function, sim: &mut StackSim, loop_top_stack: &[u32]) { @@ -2541,10 +2989,26 @@ pub fn compile_word( let mut module = Module::new(); + // A word whose stack effect is statically known gets a second, typed + // entry point; the self-recursive case is the one that pays, since the + // recursion then runs entirely in WASM values. Cross-word typed calls + // need every callee in the same module, which only CONSOLIDATE gives. + let self_id = WordId(config.base_fn_index); + let typed = config + .typed_calls + .then(|| typed_effect(body, Some(self_id), &HashMap::new())) + .flatten(); + // -- Type section -- let mut types = TypeSection::new(); types.ty().function([], []); // type 0: () -> () types.ty().function([ValType::I32], []); // type 1: (i32) -> () + if let Some((params, results)) = typed { + types.ty().function( + std::iter::repeat_n(ValType::I32, params as usize), + std::iter::repeat_n(ValType::I32, results as usize), + ); + } module.section(&types); // -- Import section -- @@ -2602,8 +3066,13 @@ pub fn compile_word( module.section(&imports); // -- Function section -- + // The wrapper stays function WORD_FUNC so the table entry, the export + // and every existing caller are unaffected; the fast entry follows it. let mut functions = FunctionSection::new(); functions.function(TYPE_VOID); + if typed.is_some() { + functions.function(TYPE_TYPED); + } module.section(&functions); // -- Export section -- @@ -2623,6 +3092,22 @@ pub fn compile_word( module.section(&elements); // -- Code section -- + if let Some(effect) = typed { + let mut callees = HashMap::new(); + callees.insert( + self_id, + TypedFn { + fn_index: TYPED_FAST_FUNC, + params: effect.0, + results: effect.1, + }, + ); + let mut code = CodeSection::new(); + code.function(&emit_typed_wrapper(effect, TYPED_FAST_FUNC)); + code.function(&emit_typed_fast(body, effect, &Rc::new(callees))); + return finish_word_module(module, name, &code, config.base_fn_index, true); + } + // Determine whether to use stack-to-local promotion let promoted = config.stack_to_local_promotion && is_promotable(body); let scratch_count = count_scratch_locals(body); @@ -2697,15 +3182,31 @@ pub fn compile_word( let mut code = CodeSection::new(); code.function(&func); - module.section(&code); + finish_word_module(module, name, &code, config.base_fn_index, false) +} + +/// Attach the code and name sections, validate, and hand back the bytes. +/// +/// The name section carries the Forth word name into wasmtime trap +/// backtraces (best-effort symbolication, WS-008); a typed word names both +/// of its entries so the innermost frame is the one that reports. +fn finish_word_module( + mut module: Module, + name: &str, + code: &CodeSection, + fn_index: u32, + typed: bool, +) -> WaferResult { + module.section(code); - // -- Name section: carries the Forth word name into wasmtime trap - // backtraces (best-effort symbolication, WS-008). let mut names = wasm_encoder::NameSection::new(); names.module(name); let mut fn_names = wasm_encoder::NameMap::new(); fn_names.append(0, "emit"); fn_names.append(WORD_FUNC, name); + if typed { + fn_names.append(TYPED_FAST_FUNC, name); + } names.functions(&fn_names); module.section(&names); @@ -2716,10 +3217,7 @@ pub fn compile_word( WaferError::ValidationError(format!("Generated WASM failed validation: {e}")) })?; - Ok(CompiledModule { - bytes, - fn_index: config.base_fn_index, - }) + Ok(CompiledModule { bytes, fn_index }) } // --------------------------------------------------------------------------- @@ -3014,8 +3512,16 @@ pub fn compile_consolidated_module( local_fn_map: &HashMap, table_size: u32, stack_guards: Option, + typed_calls: bool, ) -> WaferResult> { - compile_multi_word_module(words, local_fn_map, table_size, None, stack_guards) + compile_multi_word_module( + words, + local_fn_map, + table_size, + None, + stack_guards, + typed_calls, + ) } /// Compile an exportable WASM module with embedded memory and metadata. @@ -3029,8 +3535,16 @@ pub fn compile_exportable_module( table_size: u32, export: &ExportSections<'_>, stack_guards: Option, + typed_calls: bool, ) -> WaferResult> { - compile_multi_word_module(words, local_fn_map, table_size, Some(export), stack_guards) + compile_multi_word_module( + words, + local_fn_map, + table_size, + Some(export), + stack_guards, + typed_calls, + ) } /// Internal: build a multi-word WASM module. When `export` is `Some`, adds @@ -3041,6 +3555,7 @@ fn compile_multi_word_module( table_size: u32, export: Option<&ExportSections<'_>>, stack_guards: Option, + typed_calls: bool, ) -> WaferResult> { // Arm (or disarm) stack-guard emission for this module. GUARD_FAULT.set(stack_guards); @@ -3048,10 +3563,44 @@ fn compile_multi_word_module( let has_data = export.is_some_and(|e| !e.memory_snapshot.is_empty()); let mut module = Module::new(); + // Every word lives in this one module, so a call between two words with + // a known stack effect can pass its stack items as WASM values. The + // `() -> ()` wrappers keep their function indices and table slots, and + // the fast entries are appended after them. + let effects = if typed_calls { + typed_effects(words) + } else { + HashMap::new() + }; + let mut typed: HashMap = HashMap::new(); + let mut signatures: Vec = Vec::new(); + for (word_id, _) in words { + let Some(&effect) = effects.get(word_id) else { + continue; + }; + let fn_index = words.len() as u32 + 1 + typed.len() as u32; + typed.insert( + *word_id, + TypedFn { + fn_index, + params: effect.0, + results: effect.1, + }, + ); + signatures.push(effect); + } + let typed = Rc::new(typed); + // -- Type section -- let mut types = TypeSection::new(); types.ty().function([], []); // type 0: () -> () types.ty().function([ValType::I32], []); // type 1: (i32) -> () + for &(params, results) in &signatures { + types.ty().function( + std::iter::repeat_n(ValType::I32, params as usize), + std::iter::repeat_n(ValType::I32, results as usize), + ); + } module.section(&types); // -- Import section (same as single-word modules) -- @@ -3108,11 +3657,14 @@ fn compile_multi_word_module( ); module.section(&imports); - // -- Function section: N functions, all type void -- + // -- Function section: N `() -> ()` entries, then the typed fast ones -- let mut functions = FunctionSection::new(); for _ in words { functions.function(TYPE_VOID); } + for (i, _) in signatures.iter().enumerate() { + functions.function(TYPE_TYPED + i as u32); + } module.section(&functions); // -- Export section: export each function as "fn_0", "fn_1", etc. -- @@ -3151,7 +3703,13 @@ fn compile_multi_word_module( // -- Code section: emit each function body -- let mut code = CodeSection::new(); - for (_word_id, body) in words { + for (word_id, body) in words { + // A typed word's `() -> ()` entry is just the bridge from the memory + // stack into its fast entry; the body itself is emitted further down. + if let Some(t) = typed.get(word_id) { + code.function(&emit_typed_wrapper((t.params, t.results), t.fn_index)); + continue; + } let promoted = is_promotable(body); let scratch_count = count_scratch_locals(body); let forth_local_count = count_forth_locals(body); @@ -3223,6 +3781,12 @@ fn compile_multi_word_module( func.instruction(&Instruction::End); code.function(&func); } + // Fast entries, in the same order the function section declared them. + for (word_id, body) in words { + if let Some(t) = typed.get(word_id) { + code.function(&emit_typed_fast(body, (t.params, t.results), &typed)); + } + } module.section(&code); // -- Data section (memory snapshot for exportable modules) -- @@ -3274,6 +3838,7 @@ mod tests { table_size: 16, stack_to_local_promotion: true, stack_guards: None, + typed_calls: true, } } @@ -3499,6 +4064,7 @@ mod tests { table_size: 16, stack_to_local_promotion: true, stack_guards: None, + typed_calls: true, }; let m = compile_word("t", &[IrOp::PushI32(1)], &cfg).unwrap(); assert_eq!(m.fn_index, 7); @@ -4406,4 +4972,178 @@ mod tests { ]; assert_eq!(run_float_word(&ops), vec![pi]); } + + // =================================================================== + // Typed calling convention (fast entry + wrapper) + // =================================================================== + + /// `: FIB DUP 2 < IF EXIT THEN DUP 1- RECURSE SWAP 2 - RECURSE + ;` + fn fib_ir(self_id: WordId) -> Vec { + vec![ + IrOp::Dup, + IrOp::PushI32(2), + IrOp::Lt, + IrOp::If { + then_body: vec![IrOp::Exit], + else_body: None, + }, + IrOp::Dup, + IrOp::PushI32(1), + IrOp::Sub, + IrOp::Call(self_id), + IrOp::Swap, + IrOp::PushI32(2), + IrOp::Sub, + IrOp::Call(self_id), + IrOp::Add, + ] + } + + #[test] + fn typed_effect_solves_self_recursion() { + // The recursion makes the equation circular (2d = d), so the + // fixpoint has to settle it: FIB is ( n -- fib ). + let id = WordId(5); + assert_eq!( + typed_effect(&fib_ir(id), Some(id), &HashMap::new()), + Some((1, 1)) + ); + } + + #[test] + fn typed_effect_rejects_a_word_without_a_fixed_effect() { + // `: F 1 RECURSE ;` grows the stack by one more cell per level, so + // there is no signature to give it. + let id = WordId(5); + let body = vec![IrOp::PushI32(1), IrOp::Call(id)]; + assert_eq!(typed_effect(&body, Some(id), &HashMap::new()), None); + } + + #[test] + fn typed_effect_rejects_an_unknown_callee() { + // Nothing is known about word 9, so its caller cannot be typed + // either -- this is what keeps single-word modules to self-calls. + let body = vec![IrOp::Dup, IrOp::Call(WordId(9))]; + assert_eq!(typed_effect(&body, Some(WordId(5)), &HashMap::new()), None); + + let known = HashMap::from([(WordId(9), (1, 1))]); + assert_eq!(typed_effect(&body, Some(WordId(5)), &known), Some((1, 2))); + } + + #[test] + fn typed_effect_rejects_branches_that_disagree_on_depth() { + // ( -- ) on one side and ( -- x ) on the other: no static effect. + let body = vec![IrOp::If { + then_body: vec![IrOp::PushI32(1)], + else_body: Some(vec![]), + }]; + assert_eq!(typed_effect(&body, None, &HashMap::new()), None); + } + + #[test] + fn typed_effect_allows_an_exiting_branch_to_differ() { + // `DUP 0= IF DROP EXIT THEN 1+` -- the EXIT branch never reaches the + // join, so it does not have to agree with the fall-through. + let body = vec![ + IrOp::Dup, + IrOp::ZeroEq, + IrOp::If { + then_body: vec![IrOp::Exit], + else_body: None, + }, + IrOp::PushI32(1), + IrOp::Add, + ]; + assert_eq!(typed_effect(&body, None, &HashMap::new()), Some((1, 1))); + } + + #[test] + fn typed_effect_rejects_exits_at_different_depths() { + // One EXIT leaves a cell the other does not. + let body = vec![ + IrOp::If { + then_body: vec![IrOp::PushI32(1), IrOp::Exit], + else_body: None, + }, + IrOp::Exit, + ]; + assert_eq!(typed_effect(&body, None, &HashMap::new()), None); + } + + #[test] + fn typed_word_module_has_a_wrapper_and_a_fast_entry() { + let cfg = default_config(); + let id = WordId(cfg.base_fn_index); + let m = compile_word("FIB", &fib_ir(id), &cfg).unwrap(); + // compile_word validates, so reaching here means the two-function + // module is well-formed; check the table entry is still the wrapper. + let mut funcs = 0; + for payload in wasmparser::Parser::new(0).parse_all(&m.bytes) { + if let wasmparser::Payload::FunctionSection(s) = payload.unwrap() { + funcs = s.count(); + } + } + assert_eq!(funcs, 2, "expected wrapper + fast entry"); + } + + #[test] + fn typed_calls_can_be_turned_off() { + let cfg = CodegenConfig { + typed_calls: false, + ..default_config() + }; + let id = WordId(cfg.base_fn_index); + let m = compile_word("FIB", &fib_ir(id), &cfg).unwrap(); + let mut funcs = 0; + for payload in wasmparser::Parser::new(0).parse_all(&m.bytes) { + if let wasmparser::Payload::FunctionSection(s) = payload.unwrap() { + funcs = s.count(); + } + } + assert_eq!(funcs, 1, "memory-stack convention emits one function"); + } + + #[test] + fn typed_effects_spread_from_leaves_to_callers() { + // SQ is a leaf, SUMSQ calls it twice: the fixpoint has to settle SQ + // first, then SUMSQ becomes typeable in the next round. + let sq = WordId(1); + let sumsq = WordId(2); + let words = vec![ + (sq, vec![IrOp::Dup, IrOp::Mul]), + ( + sumsq, + vec![IrOp::Call(sq), IrOp::Swap, IrOp::Call(sq), IrOp::Add], + ), + ]; + let effects = typed_effects(&words); + assert_eq!(effects.get(&sq), Some(&(1, 1))); + assert_eq!(effects.get(&sumsq), Some(&(2, 1))); + } + + #[test] + fn mutually_recursive_words_stay_untyped() { + // Neither can be settled before the other, so both keep the + // memory-stack convention rather than looping forever. + let a = WordId(1); + let b = WordId(2); + let words = vec![(a, vec![IrOp::Call(b)]), (b, vec![IrOp::Call(a)])]; + assert!(typed_effects(&words).is_empty()); + } + + #[test] + fn consolidated_module_with_typed_calls_validates() { + let sq = WordId(1); + let sumsq = WordId(2); + let words = vec![ + (sq, vec![IrOp::Dup, IrOp::Mul]), + ( + sumsq, + vec![IrOp::Call(sq), IrOp::Swap, IrOp::Call(sq), IrOp::Add], + ), + ]; + let map = HashMap::from([(sq, 1u32), (sumsq, 2u32)]); + // compile_consolidated_module validates internally. + compile_consolidated_module(&words, &map, 16, None, true).unwrap(); + } } diff --git a/crates/core/src/config.rs b/crates/core/src/config.rs index a80e71c..db7c451 100644 --- a/crates/core/src/config.rs +++ b/crates/core/src/config.rs @@ -12,6 +12,11 @@ pub struct CodegenOpts { /// corrupting stack pointers. On by default; benchmarks and /// exported production modules turn it off. pub stack_guards: bool, + /// Compile words with a statically known stack effect to a typed entry + /// point that carries stack items in WASM values, so a call keeps them + /// in registers instead of round-tripping through the memory stack. + /// On by default; `WAFER_TYPED_CALLS=0` turns it off. + pub typed_calls: bool, } /// Master configuration for all WAFER optimizations. @@ -38,6 +43,7 @@ impl WaferConfig { codegen: CodegenOpts { stack_to_local_promotion: true, stack_guards: true, + typed_calls: true, }, } } @@ -56,6 +62,7 @@ impl WaferConfig { codegen: CodegenOpts { stack_to_local_promotion: false, stack_guards: false, + typed_calls: false, }, } } diff --git a/crates/core/src/consolidate.rs b/crates/core/src/consolidate.rs index 9c340bb..bf0e9a7 100644 --- a/crates/core/src/consolidate.rs +++ b/crates/core/src/consolidate.rs @@ -21,7 +21,7 @@ mod tests { // Empty word list should produce nothing (but we guard against this at call site) let words = vec![]; let map = HashMap::new(); - let result = compile_consolidated_module(&words, &map, 16, None); + let result = compile_consolidated_module(&words, &map, 16, None, true); // Empty is valid -- should produce a valid module with no functions assert!(result.is_ok()); } @@ -31,7 +31,7 @@ mod tests { let words = vec![(WordId(1), vec![IrOp::PushI32(42)])]; let mut map = HashMap::new(); map.insert(WordId(1), 1u32); // function index 1 (after emit import) - let result = compile_consolidated_module(&words, &map, 16, None); + let result = compile_consolidated_module(&words, &map, 16, None, true); assert!(result.is_ok()); } @@ -49,7 +49,7 @@ mod tests { map.insert(WordId(1), 1u32); map.insert(WordId(2), 2u32); map.insert(WordId(3), 3u32); - let result = compile_consolidated_module(&words, &map, 16, None); + let result = compile_consolidated_module(&words, &map, 16, None, true); assert!(result.is_ok()); } @@ -59,7 +59,7 @@ mod tests { let words = vec![(WordId(3), vec![IrOp::Call(WordId(99))])]; let mut map = HashMap::new(); map.insert(WordId(3), 1u32); - let result = compile_consolidated_module(&words, &map, 256, None); + let result = compile_consolidated_module(&words, &map, 256, None, true); assert!(result.is_ok()); } @@ -72,7 +72,7 @@ mod tests { let mut map = HashMap::new(); map.insert(WordId(1), 1u32); map.insert(WordId(2), 2u32); - let result = compile_consolidated_module(&words, &map, 16, None); + let result = compile_consolidated_module(&words, &map, 16, None, true); assert!(result.is_ok()); } @@ -95,7 +95,7 @@ mod tests { let mut map = HashMap::new(); map.insert(WordId(1), 1u32); map.insert(WordId(2), 2u32); - let result = compile_consolidated_module(&words, &map, 16, None); + let result = compile_consolidated_module(&words, &map, 16, None, true); assert!(result.is_ok()); } @@ -120,7 +120,7 @@ mod tests { let mut map = HashMap::new(); map.insert(WordId(1), 1u32); map.insert(WordId(2), 2u32); - let result = compile_consolidated_module(&words, &map, 16, None); + let result = compile_consolidated_module(&words, &map, 16, None, true); assert!(result.is_ok()); } @@ -141,7 +141,7 @@ mod tests { let mut map = HashMap::new(); map.insert(WordId(1), 1u32); map.insert(WordId(2), 2u32); - let result = compile_consolidated_module(&words, &map, 16, None); + let result = compile_consolidated_module(&words, &map, 16, None, true); assert!(result.is_ok()); } @@ -163,7 +163,7 @@ mod tests { let mut map = HashMap::new(); map.insert(WordId(1), 1u32); map.insert(WordId(2), 2u32); - let result = compile_consolidated_module(&words, &map, 16, None); + let result = compile_consolidated_module(&words, &map, 16, None, true); assert!(result.is_ok()); } } diff --git a/crates/core/src/export.rs b/crates/core/src/export.rs index 02988e4..efe6177 100644 --- a/crates/core/src/export.rs +++ b/crates/core/src/export.rs @@ -126,6 +126,7 @@ pub fn export_module( table_size, &export_sections, vm.stack_guard_param(), + vm.typed_calls(), ) .map_err(|e| anyhow::anyhow!("export codegen error: {e}"))?; diff --git a/crates/core/src/outer.rs b/crates/core/src/outer.rs index 6952bff..b15eac9 100644 --- a/crates/core/src/outer.rs +++ b/crates/core/src/outer.rs @@ -2644,6 +2644,7 @@ impl ForthVM { &local_fn_map, table_size, self.stack_guard_param(), + self.config.codegen.typed_calls, ) .map_err(|e| anyhow::anyhow!("consolidation codegen error: {e}"))?; @@ -2674,6 +2675,7 @@ impl ForthVM { &local_fn_map, table_size, self.stack_guard_param(), + self.config.codegen.typed_calls, ) .map_err(|e| anyhow::anyhow!("batch compile error: {e}"))?; @@ -2966,6 +2968,11 @@ impl ForthVM { .then_some(self.stack_fault_id) } + /// Whether words with a known stack effect get a typed entry point. + pub(crate) fn typed_calls(&self) -> bool { + self.config.codegen.typed_calls + } + /// Codegen configuration for compiling one word. fn codegen_config(&mut self, base_fn_index: u32) -> CodegenConfig { CodegenConfig { @@ -2973,6 +2980,7 @@ impl ForthVM { table_size: self.table_size(), stack_to_local_promotion: self.config.codegen.stack_to_local_promotion, stack_guards: self.stack_guard_param(), + typed_calls: self.config.codegen.typed_calls, } } @@ -8180,6 +8188,73 @@ mod tests { use super::*; use crate::runtime_native::NativeRuntime; + // -- Typed calling convention ------------------------------------- + + #[test] + fn test_typed_word_recursion_matches_the_memory_convention() { + // FIB is the shape the typed entry exists for: self-recursive with + // an early EXIT, so it is compiled as WASM values in and out. + let (stack, _) = eval( + ": FIB DUP 2 < IF EXIT THEN DUP 1- RECURSE SWAP 2 - RECURSE + ; \ + 0 FIB 1 FIB 2 FIB 10 FIB 25 FIB", + ); + assert_eq!(stack, vec![75025, 55, 1, 1, 0]); + } + + #[test] + fn test_typed_word_keeps_the_items_below_its_arguments() { + // The wrapper may only move the cells the word declared; anything + // deeper has to still be there afterwards. + let (stack, _) = eval(": SQ DUP * ; 7 8 9 SQ"); + assert_eq!(stack, vec![81, 8, 7]); + } + + #[test] + fn test_typed_word_is_reachable_through_execute() { + // EXECUTE goes through the table, which holds the `( -- )` wrapper. + let (stack, _) = eval(": SQ DUP * ; 6 ' SQ EXECUTE"); + assert_eq!(stack, vec![36]); + } + + #[test] + fn test_catch_sees_a_typed_word_underflow() { + // A typed word's stack guards live in its wrapper, where the + // arguments come off the memory stack. They still THROW -4 rather + // than corrupting the stack pointer, and CATCH still reports it. + let (stack, _) = eval(": SQ DUP * ; : CHK ['] SQ CATCH ; CHK"); + assert_eq!(stack, vec![-4]); + } + + #[test] + fn test_catch_restores_the_stack_around_a_caller_of_typed_code() { + // T contains THROW, a host word, so T itself keeps the memory + // convention and reaches SQ through its wrapper. CATCH restores the + // depth it saved across that boundary -- not the contents, so the 3 + // that SQ squared stays squared. gforth agrees: `1 2 9 5 4`. + let (stack, _) = eval(": SQ DUP * ; : T SQ 5 THROW ; : CHK 1 2 3 ['] T CATCH DEPTH ; CHK"); + assert_eq!(stack, vec![4, 5, 9, 2, 1]); + } + + #[test] + fn test_typed_word_underflow_still_throws() { + // The stack guards for a typed word live in its wrapper, where the + // arguments are taken off the memory stack. + let mut vm = ForthVM::::new().unwrap(); + vm.evaluate(": SQ DUP * ;").unwrap(); + vm.take_output(); + let err = vm.evaluate("SQ"); + assert!(err.is_err(), "empty-stack SQ should throw, got {err:?}"); + } + + #[test] + fn test_deep_typed_recursion_unwinds_cleanly() { + // 10k levels of a typed self-call: results come back in WASM values, + // so nothing is left on the memory stack afterwards. + let (stack, _) = + eval(": COUNT-DOWN DUP 0= IF EXIT THEN 1- RECURSE ; 10000 COUNT-DOWN DEPTH"); + assert_eq!(stack, vec![1, 0]); + } + fn eval(input: &str) -> (Vec, String) { let mut vm = ForthVM::::new().unwrap(); vm.evaluate(input).unwrap(); diff --git a/crates/core/tests/compliance.rs b/crates/core/tests/compliance.rs index 95978be..02cc490 100644 --- a/crates/core/tests/compliance.rs +++ b/crates/core/tests/compliance.rs @@ -344,3 +344,32 @@ fn compliance_tools() { let errors = run_suite(&mut vm, "toolstest.fth"); assert_eq!(errors, 0, "Programming-Tools: {errors} test failures"); } + +/// The Forth 2012 Core suite against consolidated code. +/// +/// `CONSOLIDATE` recompiles the whole dictionary into one WASM module, which +/// is where cross-word typed calls live: a word with a known stack effect +/// gets a fast entry taking and returning its stack items as WASM values, +/// and its `() -> ()` wrapper keeps the table slot. Nothing else covers that +/// path for correctness, so run the suite on top of it. +#[test] +fn compliance_core_after_consolidate() { + let mut vm = ForthVM::::new().expect("Failed to create ForthVM"); + let tester_path = format!("{SUITE_DIR}/tester.fr"); + let f1 = load_file(&mut vm, &tester_path); + assert_load_fails_within_baseline(&tester_path, f1); + + vm.evaluate("CONSOLIDATE").expect("CONSOLIDATE failed"); + vm.take_output(); + + let core_path = format!("{SUITE_DIR}/core.fr"); + let f2 = load_file(&mut vm, &core_path); + assert_load_fails_within_baseline(&core_path, f2); + + let _ = vm.evaluate("DECIMAL #ERRORS @"); + let errors = vm.data_stack().first().copied().unwrap_or(-1); + assert_eq!( + errors, 0, + "Core word set after CONSOLIDATE: {errors} failures" + ); +} From b8dcc021a25baa2b3d1412108b22533b9f52ca14 Mon Sep 17 00:00:00 2001 From: Oleksandr Kozachuk <201152+ok2@users.noreply.github.com> Date: Sun, 9 Aug 2026 17:25:21 +0200 Subject: [PATCH 2/3] perf(core): promote per region, promote BEGIN loops, keep loops off the memory stack Promotion was all-or-nothing per word, so one `.` or one host call put the whole body -- hot loops included -- on the memory data stack, where a loop-carried add costs 2.2 ns/iteration instead of 0.31. The stack simulator now runs over each promotable stretch of a word; BEGIN/UNTIL, BEGIN/AGAIN and BEGIN/WHILE/REPEAT join DO/LOOP as promotable when the construct is provably stack-neutral; and the inliner no longer moves a loop-bearing callee into a caller that can never be promoted. Fixes a bug the BEGIN work uncovered, present since promotion was introduced and shipped in 0.2.6: the loop fixup and the IF join copied locals one slot at a time in index order, so a body that permutes the stack lost a value -- `: C 3 4 2 0 DO SWAP LOOP . . ;` printed `4 4` where gforth prints `4 3`. Four of five benchmarks now beat sf64: Factorial 0.29x, Collatz 0.30x, NestedLoops 0.27x, GCD 0.67x. Only Fibonacci is behind, at 1.24x. Also scale GCD, Factorial and NestedLoops, which ran in 14-51 us where scatter and fixed costs dominated -- that is what exposed GCD as a loss and pointed at BEGIN. WS-014, WS-015, WS-016, WS-019. --- CHANGELOG.md | 85 +++++++- README.md | 17 +- crates/core/src/codegen.rs | 358 +++++++++++++++++++++++++++++--- crates/core/src/optimizer.rs | 67 +++++- crates/core/src/outer.rs | 82 ++++++++ crates/core/tests/comparison.rs | 24 +-- 6 files changed, 581 insertions(+), 52 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1926b5e..d56425c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,8 +27,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 convention cost the most. It now handles both. Fibonacci(25) goes from 1035 to 366 µs, 4.3x slower than `sf64` to 1.2x. - Loop-heavy benchmarks are unchanged (they were already promoted, and - already beat `sf64`). Words that keep the memory convention: anything + Loop-heavy benchmarks are unchanged by this entry — see the region + promotion below for those. Words that keep the memory convention: anything using `SP@`, `DEPTH`, `EXECUTE`, `>R`/`R>`, floats or locals; anything calling a word that is itself untyped, which in the JIT path means every call except `RECURSE`; mutually recursive words; and words whose effect is @@ -46,12 +46,93 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `WAFER_TYPED_CALLS=0` falls back to the memory-stack convention. +- **Promotion is now per region, not per word.** Stack-to-local promotion + used to be all-or-nothing: a single `.`, `CR`, `>R` or host call + anywhere in a definition put the _entire_ body on the memory data + stack, hot loops included. The stack simulator now runs over each + stretch of a word that can live in WASM locals, loading what the + region reads and writing back what it leaves, with the rest of the + word unchanged around it. + + The cliff this removes was steep. The same loop, same build: + + | `: L1 0 5000000 0 DO 1+ LOOP DROP ;` reached as | µs | ns/iter | + | ----------------------------------------------- | ----- | ------- | + | its own word | 1571 | 0.31 | + | inlined into a caller with a `.` in it (before) | 11100 | 2.22 | + | the same, after this change | 1572 | 0.31 | + + 7x, for one `i32.add`: on the memory path the accumulator is stored to + linear memory and reloaded next iteration, so the loop-carried + dependency runs through store-to-load forwarding instead of a + register. + + A region may only use `I` / `J` when the DO loops naming them are + inside the region, since the simulator resolves them against its own + loop stack. Straight-line regions have to be at least three operations + to be worth the load and store either side; a loop always is. + +- **The inliner no longer drags a loop onto the memory stack.** It + inlined any callee of eight IR operations or fewer, so a small + loop-bearing word inlined into a caller that can never be promoted + lost its registers -- an optimisation pass applying the 7x + pessimisation above. Loop-bearing callees now stay put in that case: + one call is far cheaper than a loop's worth of memory traffic. + Straight-line words still inline everywhere. + +- **`BEGIN` loops promote as well.** `BEGIN..UNTIL`, `BEGIN..AGAIN` and + `BEGIN..WHILE..REPEAT` were rejected outright by the eligibility check, + so any word built on the idiomatic Forth loop kept the memory data + stack no matter how hot it was. They are promoted now when the + construct is stack-neutral: `UNTIL` consumes exactly the flag its body + leaves, `AGAIN`'s body is neutral, and for `WHILE..REPEAT` the test and + the body balance separately -- `WHILE` leaves the loop between the two, + so a net that only added up over the pair would give the two exits + different stack shapes. Bodies containing an `EXIT` stay out, the same + rule `DO`/`LOOP` follows. `BEGIN..WHILE..WHILE..REPEAT` is still + excluded. + + GCD 994 -> 540 µs, Collatz 428 -> 185. + + Together these four entries put four of the five cross-engine + benchmarks past SwiftForth `sf64`: Factorial 0.29x, Collatz 0.30x, + NestedLoops 0.27x, GCD 0.67x. Fibonacci stays at 1.24x, being pure + call overhead with no loop to promote. + ### Fixed +- **A promoted loop or `IF` whose branch permutes the stack lost a value.** + At the bottom of a promoted loop the body's results are copied back into + the loop-top locals, and the join after a promoted `IF` copies one + branch's locals into the other's. Both did it one slot at a time in index + order, which is wrong as soon as a destination is also a later source: + `: C 3 4 2 0 DO SWAP LOOP . . ;` printed `4 4` where gforth and + SwiftForth print `4 3`, and `2 0 DO ROT LOOP` over three cells printed + `3 2 3` instead of `2 1 3`. The copies are now ordered so every source is + read before it is overwritten, with one scratch local to break a cycle. + Present since stack-to-local promotion was introduced; reachable from + any `DO` loop or `IF` whose body reorders cells it did not create. + - The Forth 2012 Core suite now also runs against consolidated code (`compliance_core_after_consolidate`). `CONSOLIDATE` had no correctness test at all before -- only benchmarks. +### Changed + +- **Three cross-engine benchmarks were too small to be measured.** GCD ran + in 14 µs, Factorial in 49 and NestedLoops in 51, where per-run scatter is + a good fraction of the total and fixed per-invocation costs in the other + engines dominate. Scaled to Factorial x100K, GCD-bench(20K) and + NestedLoops(50)x1K, all now around 0.5-1 ms. + + This changed a result rather than just steadying it: GCD looked like a + win at 0.42x of `sf64` and was in fact a loss at 1.17x. That is what + pointed at `BEGIN` loops as the remaining gap -- GCD is the one benchmark + whose loop is a `BEGIN ... WHILE ... REPEAT` -- and with those promoted it + now reads 0.67x. The regression limits, which had drifted to 3-6x looser + than the measurements they guard, were retightened to ~45% above the + current ratios. + ## [0.2.6] - 2026-08-07 ### Fixed diff --git a/README.md b/README.md index 4ba3c59..0eb62be 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ An optimizing Forth 2012 compiler targeting WebAssembly. WAFER JIT-compiles each ## Highlights - **200+ words** across 12 Forth 2012 word sets, all at **100% compliance** -- **Optimizing compiler** with 6 IR passes + stack-to-local promotion (loops + IF) + consolidation +- **Optimizing compiler** with 6 IR passes + stack-to-local promotion (per region, so a hot loop keeps its registers even inside a word that does I/O; `DO` and `BEGIN` loops alike) + consolidation - **Faster than gforth** on all benchmarks in release mode (2-10x faster) - **JIT compilation** — each `:` definition compiles to its own WASM module - **Self-recursive direct calls** — RECURSE compiles to native `call` instead of `call_indirect` @@ -85,15 +85,20 @@ reach of SwiftForth `sf64`, which compiles to native code: ``` Benchmark WAFER CONSOL gforth sf64 WAFER/gf WAFER/sf -Fibonacci(25) 378 359 3238 296 0.11x 1.21x -Factorial(12)x10K 335 320 633 183 0.51x 1.75x -GCD-bench(500) 14 15 29 31 0.48x 0.45x -NestedLoops(50) 72 69 698 207 0.10x 0.33x -Collatz(2K) 994 997 3940 668 0.25x 1.49x +Fibonacci(25) 356 361 3389 287 0.11x 1.24x +Factorial(12)x100K 479 495 6249 1650 0.08x 0.29x +GCD-bench(20K) 540 559 1801 801 0.30x 0.67x +NestedLoops(50)x1K 509 501 7023 1887 0.07x 0.27x +Collatz(2K) 185 213 3873 610 0.05x 0.30x ``` Times in microseconds. WAFER/gf < 1.0 means WAFER is faster. CONSOL = after `CONSOLIDATE`. +Two caveats on the `sf64` column. The SwiftForth build here is x86-64 running under Rosetta 2 +while WAFER and gforth are native arm64, so it is a native-vs-emulated comparison; and sf64 +uses 64-bit cells to WAFER's 32-bit. WAFER is ahead on the four loop-heavy benchmarks and +behind on Fibonacci, which is one call per node with no loop to promote. + A word whose stack effect is statically known gets a **typed entry point**: its stack items travel in and out as WASM values instead of through the memory data stack, so cranelift keeps them in registers across a call the way a native Forth keeps TOS in one. The word also keeps a `( -- )` wrapper, which is what the function diff --git a/crates/core/src/codegen.rs b/crates/core/src/codegen.rs index e2ef3fa..062f65c 100644 --- a/crates/core/src/codegen.rs +++ b/crates/core/src/codegen.rs @@ -354,6 +354,9 @@ struct EmitCtx { /// Stack of open block labels for flat forward branches (CS-ROLL'd IF/THEN). /// Used by `BranchIfFalse` to compute `br_if` depth. open_blocks: Vec, + /// First WASM local a promoted region may allocate from. Regions run one + /// after another, so they all share this pool. + region_local_base: u32, } /// Decrement the FSP global by 8 (allocate space for one f64). @@ -465,8 +468,148 @@ fn emit_float_cmp(f: &mut Function, ctx: &EmitCtx, wasm_cmp: &Instruction<'_>) { /// Emit all IR operations in `ops` into the WASM function body `f`. fn emit_body(f: &mut Function, ops: &[IrOp], ctx: &mut EmitCtx) { + let mut i = 0; + while i < ops.len() { + let run = promotable_run(&ops[i..]); + if run > 0 && region_is_worth_promoting(&ops[i..i + run]) { + emit_promoted_region(f, &ops[i..i + run], ctx); + } else { + let run = run.max(1); + for op in &ops[i..i + run] { + emit_op(f, op, ctx); + } + } + i += run.max(1); + } +} + +/// Run the stack simulator over one stretch of a word that is otherwise on +/// the memory path: load what the region reads into WASM locals, work there, +/// write the results back. +/// +/// This is what keeps a hot loop in registers inside a word that can never be +/// promoted as a whole -- one `.` or one host call used to put the entire +/// body, loops included, back on the memory data stack. +fn emit_promoted_region(f: &mut Function, ops: &[IrOp], ctx: &mut EmitCtx) { + let (preload, _) = compute_stack_needs(ops); + let mut sim = StackSim::new(ctx.region_local_base).with_move_scratch(); + emit_promoted_prologue(f, preload, &mut sim); for op in ops { - emit_op(f, op, ctx); + emit_promoted_op(f, op, &mut sim); + } + emit_promoted_epilogue(f, &mut sim); +} + +/// Length of the longest prefix of `ops` that can run as a promoted region. +fn promotable_run(ops: &[IrOp]) -> usize { + ops.iter() + .take_while(|op| { + let one = std::slice::from_ref(*op); + is_promotable_body(one, PromoteMode::Memory) && region_loop_refs_resolved(one, 0) + }) + .count() +} + +/// Is a region worth the load/store either side of it? +/// +/// A loop always is -- that is the whole point. Otherwise the prologue and +/// epilogue have to be amortised over enough operations to beat leaving them +/// on the memory stack, which costs roughly two or three accesses each. +fn region_is_worth_promoting(ops: &[IrOp]) -> bool { + ops.len() >= MIN_PROMOTED_REGION || ops.iter().any(is_loop_op) +} + +/// Smallest straight-line region worth promoting. +const MIN_PROMOTED_REGION: usize = 3; + +fn is_loop_op(op: &IrOp) -> bool { + matches!( + op, + IrOp::DoLoop { .. } + | IrOp::BeginUntil { .. } + | IrOp::BeginAgain { .. } + | IrOp::BeginWhileRepeat { .. } + | IrOp::BeginDoubleWhileRepeat { .. } + ) +} + +/// Does every `I` / `J` in `ops` refer to a DO loop that is inside `ops`? +/// +/// The promoted emitter resolves them against its own loop stack, so a region +/// that borrows the index of a loop emitted around it would read the wrong +/// local -- or, for `J` below two levels, silently emit nothing. +fn region_loop_refs_resolved(ops: &[IrOp], depth: u32) -> bool { + ops.iter().all(|op| match op { + IrOp::RFetch => depth >= 1, + IrOp::LoopJ => depth >= 2, + IrOp::DoLoop { body, .. } => region_loop_refs_resolved(body, depth + 1), + IrOp::If { + then_body, + else_body, + } => { + region_loop_refs_resolved(then_body, depth) + && else_body + .as_deref() + .is_none_or(|eb| region_loop_refs_resolved(eb, depth)) + } + IrOp::BeginUntil { body } | IrOp::BeginAgain { body } => { + region_loop_refs_resolved(body, depth) + } + IrOp::BeginWhileRepeat { test, body } => { + region_loop_refs_resolved(test, depth) && region_loop_refs_resolved(body, depth) + } + _ => true, + }) +} + +/// Locals needed by the largest single promoted region in `ops`, walking the +/// body exactly the way [`emit_body`] partitions it. +fn region_local_budget(ops: &[IrOp]) -> u32 { + let mut max = 0; + let mut i = 0; + while i < ops.len() { + let run = promotable_run(&ops[i..]); + if run > 0 && region_is_worth_promoting(&ops[i..i + run]) { + let region = &ops[i..i + run]; + let (preload, _) = compute_stack_needs(region); + max = max.max(count_promoted_locals(region, preload)); + } else { + for op in &ops[i..i + run.max(1)] { + max = max.max(region_local_budget_of_children(op)); + } + } + i += run.max(1); + } + max +} + +/// Largest region budget among an operation's nested bodies. +fn region_local_budget_of_children(op: &IrOp) -> u32 { + match op { + IrOp::If { + then_body, + else_body, + } => { + region_local_budget(then_body).max(else_body.as_deref().map_or(0, region_local_budget)) + } + IrOp::DoLoop { body, .. } | IrOp::BeginUntil { body } | IrOp::BeginAgain { body } => { + region_local_budget(body) + } + IrOp::BeginWhileRepeat { test, body } => { + region_local_budget(test).max(region_local_budget(body)) + } + IrOp::BeginDoubleWhileRepeat { + outer_test, + inner_test, + body, + after_repeat, + else_body, + } => region_local_budget(outer_test) + .max(region_local_budget(inner_test)) + .max(region_local_budget(body)) + .max(region_local_budget(after_repeat)) + .max(else_body.as_deref().map_or(0, region_local_budget)), + _ => 0, } } @@ -1281,6 +1424,30 @@ enum PromoteMode { Typed, } +/// Can this body be promoted once its calls are accounted for? +/// +/// True when the only things standing between it and the register path are +/// calls and `EXIT` -- i.e. the word either gets a typed entry or, failing +/// that, promotes region by region. False means it is stuck on the memory +/// data stack whatever happens, which is what the inliner needs to know. +pub(crate) fn promotable_modulo_calls(ops: &[IrOp]) -> bool { + is_promotable_body(ops, PromoteMode::Typed) +} + +/// Does this body contain a loop at any nesting depth? +pub(crate) fn contains_loop(ops: &[IrOp]) -> bool { + ops.iter().any(|op| { + is_loop_op(op) + || match op { + IrOp::If { + then_body, + else_body, + } => contains_loop(then_body) || else_body.as_deref().is_some_and(contains_loop), + _ => false, + } + }) +} + /// Recursive check for promotable ops. fn is_promotable_body(ops: &[IrOp], mode: PromoteMode) -> bool { let typed = mode == PromoteMode::Typed; @@ -1367,11 +1534,46 @@ fn is_promotable_body(ops: &[IrOp], mode: PromoteMode) -> bool { } } } - // BEGIN loops, BeginDoubleWhileRepeat, flat forward blocks: not promoted - IrOp::BeginUntil { .. } - | IrOp::BeginAgain { .. } - | IrOp::BeginWhileRepeat { .. } - | IrOp::BeginDoubleWhileRepeat { .. } + // BEGIN loops: the construct as a whole is stack-neutral, which is + // what lets the next iteration reuse the loop-top locals. A body + // that is not neutral has no single promoted stack shape, and an + // EXIT out of one would have to unwind the join -- the same rule + // DO/LOOP already follows. + IrOp::BeginUntil { body } | IrOp::BeginAgain { body } => { + if !is_promotable_body(body, mode) || body_has_exit(body) { + return false; + } + if !typed { + // UNTIL consumes a flag the body leaves, AGAIN consumes nothing. + let expected = i32::from(matches!(op, IrOp::BeginUntil { .. })); + let (_, body_net) = compute_stack_needs(body); + if body_net != expected { + return false; + } + } + } + IrOp::BeginWhileRepeat { test, body } => { + if !is_promotable_body(test, mode) + || !is_promotable_body(body, mode) + || body_has_exit(test) + || body_has_exit(body) + { + return false; + } + if !typed { + // WHILE leaves the loop between test and body, so the two + // have to be neutral separately: a net that only balances + // over the pair would give the two exits different shapes. + let (_, test_net) = compute_stack_needs(test); + let (_, body_net) = compute_stack_needs(body); + if test_net != 1 || body_net != 0 { + return false; + } + } + } + // BeginDoubleWhileRepeat has a promoted emitter, but one without a + // loop fixup and never exercised; flat forward blocks have none. + IrOp::BeginDoubleWhileRepeat { .. } | IrOp::Block(_) | IrOp::BranchIfFalse(_) | IrOp::EndBlock(_) @@ -1759,21 +1961,34 @@ fn compute_stack_needs_rec(ops: &[IrOp], st: &mut Needs<'_>) { IrOp::BeginUntil { body } => { let saved = st.depth; compute_stack_needs_rec(body, st); - // Body produces flag, consumed by UNTIL: net 0 for the whole construct + // Body produces the flag UNTIL consumes: net 0 for the whole + // construct, and anything else has no promoted stack shape. + if st.depth != saved + 1 { + st.consistent = false; + } st.depth = saved; } IrOp::BeginAgain { body } => { let saved = st.depth; compute_stack_needs_rec(body, st); + if st.depth != saved { + st.consistent = false; + } st.depth = saved; } IrOp::BeginWhileRepeat { test, body } => { let saved = st.depth; compute_stack_needs_rec(test, st); - // WHILE consumes flag + // WHILE consumes the flag, and leaves the loop right here, so + // test and body have to balance separately rather than as a pair. + if st.depth != saved + 1 { + st.consistent = false; + } st.depth -= 1; compute_stack_needs_rec(body, st); - // Whole construct is stack-neutral + if st.depth != saved { + st.consistent = false; + } st.depth = saved; } IrOp::BeginDoubleWhileRepeat { @@ -1821,7 +2036,8 @@ fn compute_stack_needs_rec(ops: &[IrOp], st: &mut Needs<'_>) { /// DSP and scratch locals). This is an upper bound -- we allocate a fresh /// local for each value-producing operation. fn count_promoted_locals(ops: &[IrOp], preload: u32) -> u32 { - let mut count = preload; + // +1 for the simulator's cycle-breaking scratch (`with_move_scratch`). + let mut count = preload + 1; count_promoted_locals_body(ops, &mut count); count } @@ -1913,6 +2129,10 @@ struct StackSim { /// True once the code emitted so far cannot fall through (an `EXIT` ran). /// The join after an `IF` uses it to take the surviving branch's state. diverged: bool, + /// Spare local reserved for breaking a cycle in `emit_parallel_move`. + /// Reserved before any value local so that the `IF` join, which rewinds + /// `next_local` for the else arm, can never hand it out twice. + move_scratch: Option, } /// What the typed emitter needs beyond the simulator itself. @@ -1932,13 +2152,22 @@ impl StackSim { loop_index_stack: Vec::new(), typed: None, diverged: false, + move_scratch: None, } } + /// Reserve the cycle-breaking local. Every simulator that emits promoted + /// operations needs this; the typed wrapper, which only shuffles the + /// memory stack, does not. `count_promoted_locals` budgets for it. + fn with_move_scratch(mut self) -> Self { + self.move_scratch = Some(self.alloc()); + self + } + /// Simulator for a typed fast entry: params occupy locals `0..params`, /// so fresh locals start above them. fn new_typed(params: u32, results: u32, callees: &Rc>) -> Self { - let mut sim = Self::new(params); + let mut sim = Self::new(params).with_move_scratch(); sim.stack = (0..params).collect(); sim.typed = Some(TypedCtx { results, @@ -2313,14 +2542,10 @@ fn emit_promoted_op(f: &mut Function, op: &IrOp, sim: &mut StackSim) { // join state is the else state, already in sim.stack } else { if !else_diverged { - let else_stack = &sim.stack; - let min_len = then_stack.len().min(else_stack.len()); - for i in 0..min_len { - if then_stack[i] != else_stack[i] { - f.instruction(&Instruction::LocalGet(else_stack[i])); - f.instruction(&Instruction::LocalSet(then_stack[i])); - } - } + let min_len = then_stack.len().min(sim.stack.len()); + let dsts = then_stack[..min_len].to_vec(); + let srcs = sim.stack[..min_len].to_vec(); + emit_parallel_move(f, sim, &dsts, &srcs); } sim.stack = then_stack; } @@ -2438,6 +2663,12 @@ fn emit_promoted_op(f: &mut Function, op: &IrOp, sim: &mut StackSim) { let cond = sim.pop(); f.instruction(&Instruction::LocalGet(cond)); f.instruction(&Instruction::I32Eqz); + // WHILE leaves the loop here, so the loop-top locals have to hold + // the right values on the way out too -- a test that permutes + // (`BEGIN SWAP DUP WHILE`) would otherwise leave them crossed. + // The flag is already on the operand stack, so moving locals + // between it and the `br_if` is safe. + emit_promoted_loop_fixup(f, sim, &loop_top_stack); f.instruction(&Instruction::BrIf(1)); // break to outer block emit_promoted_body(f, body, sim); @@ -2623,16 +2854,69 @@ fn emit_promoted_loop_fixup(f: &mut Function, sim: &mut StackSim, loop_top_stack sim.stack.len(), loop_top_stack.len() ); - for (i, &top_local) in loop_top_stack.iter().enumerate() { - if sim.stack[i] != top_local { - f.instruction(&Instruction::LocalGet(sim.stack[i])); - f.instruction(&Instruction::LocalSet(top_local)); - } - } + let srcs = sim.stack.clone(); + emit_parallel_move(f, sim, loop_top_stack, &srcs); // Reset sim to loop-top state sim.stack = loop_top_stack.to_vec(); } +/// Emit `dsts[i] := srcs[i]` for every `i`, all at once. +/// +/// Copying them in index order is wrong as soon as a destination is also a +/// later source: `BEGIN ... SWAP ... UNTIL` would write the top into the +/// second slot and then read that slot back, so both end up holding the same +/// value. The moves are ordered so that every source is read before it is +/// overwritten, and a cycle -- which has no such order -- is broken by +/// stashing one source in `sim.move_scratch`. +/// +/// One scratch local is enough for any number of cycles: the loop only breaks +/// a new cycle once nothing else can be emitted, and by then the previous +/// cycle has drained and released it. +fn emit_parallel_move(f: &mut Function, sim: &mut StackSim, dsts: &[u32], srcs: &[u32]) { + let mut pending: Vec<(u32, u32)> = dsts + .iter() + .zip(srcs) + .filter(|(d, s)| d != s) + .map(|(d, s)| (*d, *s)) + .collect(); + + while !pending.is_empty() { + let before = pending.len(); + let mut i = 0; + while i < pending.len() { + let (dst, src) = pending[i]; + // Safe to write `dst` now only if nothing still has to read it. + if pending + .iter() + .enumerate() + .all(|(j, (_, s))| j == i || *s != dst) + { + f.instruction(&Instruction::LocalGet(src)); + f.instruction(&Instruction::LocalSet(dst)); + pending.remove(i); + } else { + i += 1; + } + } + if pending.len() == before { + // Everything left is a cycle. Lift one source out of it, which + // frees its local and turns the cycle into a chain. + let (dst, src) = pending.remove(0); + let tmp = sim + .move_scratch + .expect("promoted simulator without a move scratch local"); + f.instruction(&Instruction::LocalGet(src)); + f.instruction(&Instruction::LocalSet(tmp)); + for p in &mut pending { + if p.1 == src { + p.1 = tmp; + } + } + pending.push((dst, tmp)); + } + } +} + /// Emit a promoted binary operation (commutative). fn emit_promoted_binary(f: &mut Function, sim: &mut StackSim, op: &Instruction<'_>) { let b = sim.pop(); @@ -3114,13 +3398,20 @@ pub fn compile_word( let forth_local_count = count_forth_locals(body); let loop_depth = count_loop_depth(body); let loop_local_count = loop_depth * 2; // 2 locals per nesting level (index, limit) + // Words on the memory path still promote what they can, region by + // region, so they need a pool of locals for that on top of everything else. + let region_locals = if promoted { + 0 + } else { + region_local_budget(body) + }; let num_locals = if promoted { let (preload, _) = compute_stack_needs(body); let promoted_count = count_promoted_locals(body, preload); // 1 (cached DSP) + promoted locals (scratch locals not needed in promoted path) 1 + promoted_count + forth_local_count + loop_local_count } else { - 1 + scratch_count + forth_local_count + loop_local_count + 1 + scratch_count + forth_local_count + loop_local_count + region_locals }; let forth_f_local_count = count_forth_f_locals(body); // F: locals need f64 storage, which also implies the f64 scratch pair. @@ -3143,6 +3434,7 @@ pub fn compile_word( 1 + scratch_count }; let loop_local_base = forth_local_base + forth_local_count; + let region_local_base = loop_local_base + loop_local_count; // f64 scratch pair first (indices num_locals, num_locals+1), then F: locals. let forth_f_local_base = num_locals + 2; let mut ctx = EmitCtx { @@ -3155,6 +3447,7 @@ pub fn compile_word( fast_loop_depth: 0, self_word_id: Some(WordId(config.base_fn_index)), open_blocks: Vec::new(), + region_local_base, }; // Prologue: cache $dsp global into local 0 @@ -3164,7 +3457,7 @@ pub fn compile_word( if promoted { let (preload, _) = compute_stack_needs(body); let first_promoted = SCRATCH_BASE; // promoted locals start right after cached_dsp - let mut sim = StackSim::new(first_promoted); + let mut sim = StackSim::new(first_promoted).with_move_scratch(); emit_promoted_prologue(&mut func, preload, &mut sim); for op in body { emit_promoted_op(&mut func, op, &mut sim); @@ -3715,12 +4008,17 @@ fn compile_multi_word_module( let forth_local_count = count_forth_locals(body); let loop_depth = count_loop_depth(body); let loop_local_count = loop_depth * 2; + let region_locals = if promoted { + 0 + } else { + region_local_budget(body) + }; let num_locals = if promoted { let (preload, _) = compute_stack_needs(body); let promoted_count = count_promoted_locals(body, preload); 1 + promoted_count + forth_local_count + loop_local_count } else { - 1 + scratch_count + forth_local_count + loop_local_count + 1 + scratch_count + forth_local_count + loop_local_count + region_locals }; let forth_f_local_count = count_forth_f_locals(body); let has_floats = needs_f64_locals(body) || forth_f_local_count > 0; @@ -3742,6 +4040,7 @@ fn compile_multi_word_module( 1 + scratch_count }; let loop_local_base = forth_local_base + forth_local_count; + let region_local_base = loop_local_base + loop_local_count; let forth_f_local_base = num_locals + 2; let mut ctx = EmitCtx { f64_local_0: num_locals, @@ -3753,6 +4052,7 @@ fn compile_multi_word_module( fast_loop_depth: 0, self_word_id: None, // consolidated module uses direct calls via local_fn_map open_blocks: Vec::new(), + region_local_base, }; // Prologue: cache $dsp global into local 0 @@ -3763,7 +4063,7 @@ fn compile_multi_word_module( // Use stack-to-local promotion (same as compile_word path) let (preload, _) = compute_stack_needs(body); let first_promoted = SCRATCH_BASE; - let mut sim = StackSim::new(first_promoted); + let mut sim = StackSim::new(first_promoted).with_move_scratch(); emit_promoted_prologue(&mut func, preload, &mut sim); for op in body { emit_promoted_op(&mut func, op, &mut sim); diff --git a/crates/core/src/optimizer.rs b/crates/core/src/optimizer.rs index 291d420..e4c1da9 100644 --- a/crates/core/src/optimizer.rs +++ b/crates/core/src/optimizer.rs @@ -53,7 +53,12 @@ pub fn optimize( // Phase 2: inline then simplify again if config.inline { - ir = inline(ir, bodies, 8); + // A caller that can never leave the memory data stack would drag an + // inlined loop down with it, so leave those callees where they are: + // as their own word the loop keeps its registers, and one call is far + // cheaper than a loop's worth of memory traffic. + let keep_loops_out = !crate::codegen::promotable_modulo_calls(&ir); + ir = inline(ir, bodies, 8, keep_loops_out); } if config.peephole { ir = peephole(ir); @@ -496,7 +501,12 @@ fn dce(ops: Vec) -> Vec { /// Inline small word bodies: replaces `Call(id)` with the word's IR body /// if the body is small enough and not recursive. -fn inline(ops: Vec, bodies: &HashMap>, max_size: usize) -> Vec { +fn inline( + ops: Vec, + bodies: &HashMap>, + max_size: usize, + keep_loops_out: bool, +) -> Vec { let mut out = Vec::new(); for op in ops { match &op { @@ -505,6 +515,7 @@ fn inline(ops: Vec, bodies: &HashMap>, max_size: usize) && body.len() <= max_size && !contains_call_to(body, *id) && !contains_exit(body) + && !(keep_loops_out && crate::codegen::contains_loop(body)) { // Inline the body, recursively converting TailCall back to Call // (tail position in the callee is not tail position in the caller). @@ -517,7 +528,7 @@ fn inline(ops: Vec, bodies: &HashMap>, max_size: usize) } _ => { out.push(apply_to_bodies(op, &|inner| { - inline(inner, bodies, max_size) + inline(inner, bodies, max_size, keep_loops_out) })); } } @@ -1012,4 +1023,54 @@ mod tests { let result = optimize(vec![IrOp::Call(WordId(5))], &config, &bodies); assert_eq!(result, vec![IrOp::Call(WordId(5))]); } + + #[test] + fn keeps_a_loop_out_of_a_caller_stuck_on_the_memory_stack() { + // The caller has a `.`, so it can never leave the memory data stack. + // Inlining the loop would drag it down too; as its own word the loop + // keeps its registers and the caller just pays one call. + let mut bodies = HashMap::new(); + bodies.insert( + WordId(5), + vec![IrOp::DoLoop { + body: vec![IrOp::PushI32(1), IrOp::Add], + is_plus_loop: false, + }], + ); + let result = opt_with_inline(vec![IrOp::Call(WordId(5)), IrOp::Dot], &bodies); + assert!( + matches!(result.first(), Some(IrOp::Call(WordId(5)))), + "loop should not have been inlined, got {result:?}" + ); + } + + #[test] + fn still_inlines_a_loop_into_a_caller_that_can_be_promoted() { + let mut bodies = HashMap::new(); + bodies.insert( + WordId(5), + vec![IrOp::DoLoop { + body: vec![IrOp::PushI32(1), IrOp::Add], + is_plus_loop: false, + }], + ); + let result = opt_with_inline(vec![IrOp::Call(WordId(5)), IrOp::Dup], &bodies); + assert!( + !result.iter().any(|op| matches!(op, IrOp::Call(_))), + "loop should have been inlined, got {result:?}" + ); + } + + #[test] + fn still_inlines_straight_line_words_anywhere() { + // Only loops are held back; a small straight-line word is still + // better off inlined even into an unpromotable caller. + let mut bodies = HashMap::new(); + bodies.insert(WordId(5), vec![IrOp::Dup, IrOp::Mul]); + let result = opt_with_inline(vec![IrOp::Call(WordId(5)), IrOp::Dot], &bodies); + assert!( + !result.iter().any(|op| matches!(op, IrOp::Call(_))), + "straight-line word should still inline, got {result:?}" + ); + } } diff --git a/crates/core/src/outer.rs b/crates/core/src/outer.rs index b15eac9..8cf77e7 100644 --- a/crates/core/src/outer.rs +++ b/crates/core/src/outer.rs @@ -8255,6 +8255,88 @@ mod tests { assert_eq!(stack, vec![1, 0]); } + // -- Region promotion (a hot loop inside an unpromotable word) ----- + + #[test] + fn test_loop_in_an_unpromotable_word_still_computes() { + // `.` keeps MIXED off the register path as a whole, but the loop + // inside it is promoted as its own region. Values checked against + // gforth 0.7.3. + assert_eq!( + eval_output(": MIXED 0 1000 0 DO 1+ LOOP . ; MIXED"), + "1000 " + ); + } + + #[test] + fn test_j_in_a_promoted_region_reads_the_right_loop() { + // A region may only use `I` / `J` when the DO loops they name are + // inside the region itself -- otherwise the simulator resolves them + // against its own empty loop stack. gforth prints 9. + assert_eq!( + eval_output(": JT 0 3 0 DO 3 0 DO J + LOOP LOOP . ; JT"), + "9 " + ); + assert_eq!(eval_output(": IT 0 5 0 DO I + LOOP . ; IT"), "10 "); + } + + #[test] + fn test_promoted_loop_body_that_permutes_the_stack() { + // The values a loop body leaves have to reach the loop-top locals all + // at once. Copying them in index order writes the top into the second + // slot and then reads that slot back, so both come out equal -- this + // printed "4 4" and "3 2 3" before. gforth: "4 3" and "2 1 3". + assert_eq!(eval_output(": C 3 4 2 0 DO SWAP LOOP . . ; C"), "4 3 "); + assert_eq!(eval_output(": D 1 2 3 2 0 DO ROT LOOP . . . ; D"), "2 1 3 "); + } + + #[test] + fn test_promoted_begin_loops() { + // BEGIN loops promote too, so these run entirely in locals. + assert_eq!( + eval_output(": GCD BEGIN DUP WHILE TUCK MOD REPEAT DROP . ; 1071 462 GCD"), + "21 " + ); + assert_eq!( + eval_output(": CD BEGIN 1 - DUP 0= UNTIL DROP 42 . ; 5 CD"), + "42 " + ); + // A WHILE test that permutes: the loop is left between test and body, + // so that exit needs the loop-top locals straightened out as well. + assert_eq!( + eval_output(": W BEGIN SWAP DUP WHILE 1 - SWAP REPEAT . . ; 9 3 W"), + "0 3 " + ); + } + + #[test] + fn test_begin_loop_with_an_unbalanced_body_is_not_promoted() { + // `BEGIN DUP 1+ SWAP DUP 5 > UNTIL` leaves one extra cell per pass, so + // there is no fixed promoted stack shape. It has to keep working. + assert_eq!( + eval_output(": U 0 BEGIN 1 + DUP DUP 3 > UNTIL DROP . . . . ; U"), + "4 3 2 1 " + ); + } + + #[test] + fn test_several_regions_in_one_word() { + // Two loops separated by a `.`: each is its own region, and the + // stack has to survive the hand-off through memory between them. + assert_eq!( + eval_output(": M2 0 10 0 DO I + LOOP DUP . 5 0 DO 1+ LOOP . ; M2"), + "45 50 " + ); + } + + #[test] + fn test_region_hands_results_back_to_the_memory_stack() { + // The region computes in locals; what it leaves has to be visible to + // the interpreter afterwards. + let (stack, _) = eval(": R 7 4 0 DO 1+ LOOP ; 100 R"); + assert_eq!(stack, vec![11, 100]); + } + fn eval(input: &str) -> (Vec, String) { let mut vm = ForthVM::::new().unwrap(); vm.evaluate(input).unwrap(); diff --git a/crates/core/tests/comparison.rs b/crates/core/tests/comparison.rs index baa1470..b98d061 100644 --- a/crates/core/tests/comparison.rs +++ b/crates/core/tests/comparison.rs @@ -746,37 +746,37 @@ fn perf_benchmarks() -> Vec { verify: "25 FIB", expected: 75025, samples: 5, - max_ratio: 0.65, + max_ratio: 0.17, }, PerfBenchmark { - name: "Factorial(12)x10K", + name: "Factorial(12)x100K", define: ": FACT 1 SWAP 1+ 1 ?DO I * LOOP ; \ - : FACT-BENCH 10000 0 DO 12 FACT DROP LOOP ;", + : FACT-BENCH 100000 0 DO 12 FACT DROP LOOP ;", run_code: "FACT-BENCH", verify: "12 FACT", expected: 479001600, samples: 5, - max_ratio: 0.75, + max_ratio: 0.12, }, PerfBenchmark { - name: "GCD-bench(500)", + name: "GCD-bench(20K)", define: ": GCD BEGIN DUP WHILE TUCK MOD REPEAT DROP ; \ : GCD-BENCH 0 DO 10000 I 1+ GCD DROP LOOP ;", - run_code: "500 GCD-BENCH", + run_code: "20000 GCD-BENCH", verify: "48 36 GCD", expected: 12, samples: 5, - max_ratio: 0.70, + max_ratio: 0.45, }, PerfBenchmark { - name: "NestedLoops(50)", + name: "NestedLoops(50)x1K", define: ": NESTED 0 SWAP 0 DO I 0 ?DO I J + DROP LOOP LOOP ; \ - : NESTED-BENCH 100 0 DO 50 NESTED DROP LOOP ;", + : NESTED-BENCH 1000 0 DO 50 NESTED DROP LOOP ;", run_code: "NESTED-BENCH", verify: "5 NESTED", expected: 0, - samples: 3, - max_ratio: 0.20, + samples: 5, + max_ratio: 0.11, }, PerfBenchmark { name: "Collatz(2K)", @@ -788,7 +788,7 @@ fn perf_benchmarks() -> Vec { verify: "27 COLLATZ", expected: 111, samples: 3, - max_ratio: 0.45, + max_ratio: 0.08, }, ] } From 3bb613ece034aa2ef9cc461152f712b4aebf80c6 Mon Sep 17 00:00:00 2001 From: Oleksandr Kozachuk <201152+ok2@users.noreply.github.com> Date: Sun, 9 Aug 2026 17:36:52 +0200 Subject: [PATCH 3/3] release: 0.2.7 Version bump plus a doc sweep: the benchmark tables in README and docs/OPTIMIZATIONS.md were still from before the typed calling convention, OPTIMIZATIONS listed BEGIN loop promotion as not started, and the subroutine-threading section of docs/WAFER.md described the memory ABI as the only one. --- CHANGELOG.md | 3 +- CLAUDE.md | 4 +-- Cargo.lock | 6 ++-- Cargo.toml | 2 +- README.md | 4 +-- crates/cli/Cargo.toml | 2 +- crates/web/Cargo.toml | 2 +- docs/OPTIMIZATIONS.md | 66 ++++++++++++++++++++++++++++++++----------- docs/WAFER.md | 6 ++-- 9 files changed, 65 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d56425c..f9a7e07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ All notable changes to WAFER are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [0.2.7] - 2026-08-09 ### Added @@ -340,6 +340,7 @@ compliance suite, `CONSOLIDATE` whole-program recompilation, `wafer build` AOT export (WASM / native / JS loader), browser REPL, SHA-1/256/512 words, and cross-engine benchmark lanes against gforth and SwiftForth. +[0.2.7]: https://github.com/ok2/wafer/compare/v0.2.6...v0.2.7 [0.2.1]: https://github.com/ok2/wafer/compare/v0.2.0...v0.2.1 [0.2.0]: https://github.com/ok2/wafer/compare/v0.1.0...v0.2.0 [0.1.0]: https://github.com/ok2/wafer/releases/tag/v0.1.0 diff --git a/CLAUDE.md b/CLAUDE.md index 5eeaa9e..5de9fd4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ ## What is WAFER? -WAFER (WebAssembly Forth Engine in Rust) is an optimizing Forth 2012 compiler targeting WebAssembly. Currently a working Forth system with 200+ words, JIT compilation, 12 word sets at 100% compliance, and a full optimization pipeline (peephole, constant folding, inlining, strength reduction, DCE, tail calls, stack-to-local promotion with loop/IF support, self-recursive direct calls, consolidation). Beats gforth on all benchmarks in release mode. Includes a browser-based REPL via wasm-pack. +WAFER (WebAssembly Forth Engine in Rust) is an optimizing Forth 2012 compiler targeting WebAssembly. Currently a working Forth system with 200+ words, JIT compilation, 12 word sets at 100% compliance, and a full optimization pipeline (peephole, constant folding, inlining, strength reduction, DCE, tail calls, per-region stack-to-local promotion with DO/BEGIN loop and IF support, self-recursive direct calls, a typed calling convention for words with a known stack effect, consolidation). Beats gforth on all benchmarks in release mode, and SwiftForth `sf64` on four of five. Includes a browser-based REPL via wasm-pack. ## Architecture @@ -79,7 +79,7 @@ Handle in `interpret_token_immediate()` or `compile_token()` as a special case. ## Testing -- Run `cargo test --workspace` before committing (currently 562 unit + 1 benchmark + 11 compliance + 9 comparison + 5 crypto) +- Run `cargo test --workspace` before committing (currently 601 unit + 1 benchmark + 12 compliance + 9 comparison + 5 crypto) - Forth 2012 compliance: `cargo test -p wafer-core --test compliance` - Cross-engine comparison (vs gforth): `cargo test -p wafer-core --test comparison` - Performance benchmarks (release mode): `cargo test -p wafer-core --test comparison -- --nocapture --ignored` diff --git a/Cargo.lock b/Cargo.lock index 45f2cf5..2906605 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1589,7 +1589,7 @@ checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" [[package]] name = "wafer" -version = "0.2.6" +version = "0.2.7" dependencies = [ "anyhow", "clap", @@ -1600,7 +1600,7 @@ dependencies = [ [[package]] name = "wafer-core" -version = "0.2.6" +version = "0.2.7" dependencies = [ "anyhow", "insta", @@ -1615,7 +1615,7 @@ dependencies = [ [[package]] name = "wafer-web" -version = "0.2.6" +version = "0.2.7" dependencies = [ "anyhow", "js-sys", diff --git a/Cargo.toml b/Cargo.toml index 440d1cc..5cda82a 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ members = ["crates/*"] resolver = "2" [workspace.package] -version = "0.2.6" +version = "0.2.7" edition = "2024" license = "MIT OR Apache-2.0" repository = "https://github.com/ok2/wafer" diff --git a/README.md b/README.md index 0eb62be..76eccc4 100644 --- a/README.md +++ b/README.md @@ -109,7 +109,7 @@ Call-heavy code is what this pays for -- Fibonacci went from 4.3x slower than `s ## Testing ```bash -# All tests (~620 currently passing) +# All tests (~628 currently passing) cargo test --workspace # Forth 2012 compliance suite @@ -142,7 +142,7 @@ Forth Source -> Outer Interpreter -> IR -> [Optimize] -> WASM Codegen (wasm-enco - `WebRuntime` — browser WebAssembly API via js-sys, for the browser REPL - **Subroutine threading** via WASM function tables (`call_indirect` for cross-word, direct `call` for self-recursion) - **JIT mode**: each new word compiles to a separate WASM module linked to shared memory/globals/table -- **IR-based pipeline** with 6 optimization passes (peephole, constant folding, strength reduction, DCE, tail call detection, inlining) plus stack-to-local promotion (with loop and IF/ELSE support), DO/LOOP index locals, and consolidation +- **IR-based pipeline** with 6 optimization passes (peephole, constant folding, strength reduction, DCE, tail call detection, inlining) plus per-region stack-to-local promotion (DO and BEGIN loops, IF/ELSE), DO/LOOP index locals, typed entry points for words with a known stack effect, and consolidation - **Dictionary**: linked-list word headers in simulated linear memory ## Project Structure diff --git a/crates/cli/Cargo.toml b/crates/cli/Cargo.toml index 7a4e652..89050bc 100644 --- a/crates/cli/Cargo.toml +++ b/crates/cli/Cargo.toml @@ -9,7 +9,7 @@ license.workspace = true workspace = true [dependencies] -wafer-core = { path = "../core", version = "0.2.6" } +wafer-core = { path = "../core", version = "0.2.7" } wasmtime = { workspace = true } anyhow = { workspace = true } clap = { version = "4", features = ["derive"] } diff --git a/crates/web/Cargo.toml b/crates/web/Cargo.toml index 71de7b5..d237ff4 100644 --- a/crates/web/Cargo.toml +++ b/crates/web/Cargo.toml @@ -12,7 +12,7 @@ workspace = true crate-type = ["cdylib", "rlib"] [dependencies] -wafer-core = { path = "../core", version = "0.2.6", default-features = false, features = ["crypto"] } +wafer-core = { path = "../core", version = "0.2.7", default-features = false, features = ["crypto"] } wasm-bindgen = "0.2" js-sys = "0.3" send_wrapper = { workspace = true } diff --git a/docs/OPTIMIZATIONS.md b/docs/OPTIMIZATIONS.md index 33c5709..2cedde8 100644 --- a/docs/OPTIMIZATIONS.md +++ b/docs/OPTIMIZATIONS.md @@ -14,7 +14,7 @@ This document describes every optimization that makes sense for WAFER, why it ma | # | Optimization | Level | Status | Impact | | -- | -------------------------- | ------------ | ----------- | ------- | -| 1 | Stack-to-Local Promotion | Codegen | Phase 2 | Highest | +| 1 | Stack-to-Local Promotion | Codegen | Phase 4 | Highest | | 2 | Peephole Optimization | IR pass | Done | High | | 3 | Constant Folding | IR pass | Done | High | | 4 | Inlining | IR pass | Done | High | @@ -29,12 +29,18 @@ This document describes every optimization that makes sense for WAFER, why it ma | 13 | Startup Batching | Architecture | Done | Low | | 14 | Self-Recursive Direct Call | Codegen | Done | High | | 15 | Float / Double-Cell | Codegen | Not started | Future | +| 16 | Typed Calling Convention | Codegen | Done | Highest | ## 1. Stack-to-Local Promotion -**Status: Phase 2 done.** Words with straight-line code, DO/LOOP, and IF/ELSE use WASM locals instead of memory stack. Stack manipulation ops (Swap, Rot, Nip, Tuck, Dup, Drop) emit zero WASM instructions. Loop index/limit kept in WASM locals (zero return stack traffic). Switchable via `WaferConfig::codegen.stack_to_local_promotion`. +**Status: Phase 4 done.** Straight-line code, DO/LOOP, IF/ELSE and the BEGIN loop family use WASM locals instead of the memory stack, per region rather than per word. Stack manipulation ops (Swap, Rot, Nip, Tuck, Dup, Drop) emit zero WASM instructions. Loop index/limit stay in WASM locals (zero return stack traffic). Switchable via `WaferConfig::codegen.stack_to_local_promotion`. -Phase 1 covered straight-line code only. Phase 2 extends to DO/LOOP (with stack-neutrality check) and IF/ELSE/THEN (with equal-branch-effect check). BEGIN loops and BeginDoubleWhileRepeat are not yet promoted. +- **Phase 1** — straight-line code. +- **Phase 2** — DO/LOOP (stack-neutrality check) and IF/ELSE/THEN (equal-branch-effect check). +- **Phase 3** — _per region instead of per word_. Promotion used to be all-or-nothing: one `.`, `CR`, `>R` or host call anywhere in a definition put the entire body on the memory stack, hot loops included, which costs 2.2 ns per loop-carried add instead of 0.31 — the accumulator round-trips through store-to-load forwarding rather than staying in a register. `emit_body` now partitions a body into maximal promotable stretches and runs the simulator over each, loading what a region reads and writing back what it leaves. A region may only use `I` / `J` when the DO loops naming them are inside it, and a straight-line region needs at least three operations to pay for its prologue and epilogue; a loop always does. +- **Phase 4** — `BEGIN..UNTIL`, `BEGIN..AGAIN` and `BEGIN..WHILE..REPEAT`, when the construct is provably stack-neutral: UNTIL's body nets +1 (the flag it consumes), AGAIN's nets 0, and for WHILE..REPEAT the test and the body must balance _separately_, because WHILE leaves the loop between them and a net that only added up over the pair would give the two exits different stack shapes. Bodies containing an `EXIT` stay out, the same rule DO/LOOP follows. + +Still not promoted: `BeginDoubleWhileRepeat`, `>R`/`R>`, floats, `{: :}` locals, `SP@`/`DEPTH`/`EXECUTE`, and the flat forward-block IR ops. ### The Problem @@ -452,33 +458,59 @@ Fibonacci(25) with ~243K recursive calls: The optimization is implemented in `emit_op` for `IrOp::Call`: when `ctx.self_word_id == Some(word_id)`, emit `call WORD_FUNC` (function index 1 in the word's own module). The `self_word_id` is derived from `CodegenConfig::base_fn_index`. +The numbers above are the state before section 16: they measure the call instruction, and what dominated turned out to be the calling _convention_ around it. A self-recursive word that is also typed now calls its own fast entry instead, and Fibonacci(25) is 356 microseconds rather than 1.6 ms. + ## 15. Float and Double-Cell Stack **Status: Not started.** `PushI64` and `PushF64` exist as IR ops but are stubs in codegen. Float stack operations are currently all host functions. The float stack lives in its own memory region (0x2540--0x2D40). Float operations will have the same memory-based overhead as integer operations, but worse: `f64` values are 8 bytes, doubling the memory traffic per push/pop. Stack-to-local promotion (section 1) is even more impactful for floats because WASM has native `f64` locals and operand stack support. +## 16. Typed Calling Convention + +**Status: Done.** A word whose stack effect is statically known compiles to two entry points: a fast one with signature `(i32 x p) -> (i32 x q)`, carrying its stack items as WASM values, and the usual `( -- )` wrapper that moves those items on and off the memory data stack. The wrapper keeps the function-table slot, so `EXECUTE`, the outer interpreter, host words and `CATCH` see exactly the ABI they saw before; only direct calls inside a module take the fast entry. `WAFER_TYPED_CALLS=0` falls back. + +### The Problem + +This is what the SwiftForth gap was made of. sf64 keeps TOS in `RBX` and the stack pointer in `RBP`, and both survive a `CALL` untouched, so its `FIB` is 16 instructions and about 7 memory touches per node. WAFER kept the whole stack in linear memory and flushed its cached `$dsp` to an imported global before every call: about 36 touches. Section 1's simulator, which already promoted loop and `IF` bodies into locals, refused any body containing a call or an `EXIT` -- exactly the words where the convention cost the most. + +### The Effect Fixpoint + +Self-recursion makes the stack-effect equation circular (`d = k + m*d`), so the effect is solved by iterating a guess until it reproduces itself: `FIB` settles on `(1,1)` in two rounds, while `: F 1 RECURSE ;` never settles and stays untyped. `CONSOLIDATE` extends this across words, since it puts them all in one module: the effects are solved from the leaves outward, and 105 of 187 words in a booted dictionary end up typed. + +### Impact + +Fibonacci(25) went from 1035 to 366 microseconds, 4.3x slower than `sf64` to 1.2x. Stack guards became nearly free as a side effect -- they hang off the memory-stack push/pop choke points, and a typed word barely has any -- so the default guards-on configuration that the REPL and the web build use went from 1631 to 365 microseconds on the same benchmark. + +Untyped by design: anything using `SP@`, `DEPTH`, `EXECUTE`, `>R`/`R>`, floats or locals; anything calling a word that is itself untyped, which in the JIT path means every call except `RECURSE`; mutually recursive words; and words whose effect is not static -- branches that disagree on depth, `EXIT` at the wrong depth, a non-neutral loop body, or a recursion that grows the stack per level. + ## Current Performance vs Gforth All optimizations enabled, release mode, measured with UTIME: ``` -Benchmark WAFER CONSOL gforth WAFER/gf -Fibonacci(25) 1629 1535 3422 0.45x -Factorial(12)x10K 340 339 638 0.53x -GCD-bench(500) 18 15 30 0.50x -NestedLoops(50) 84 73 720 0.10x -Collatz(2K) 1212 1202 3914 0.31x +Benchmark WAFER CONSOL gforth sf64 WAFER/gf WAFER/sf +Fibonacci(25) 356 361 3389 287 0.11x 1.24x +Factorial(12)x100K 479 495 6249 1650 0.08x 0.29x +GCD-bench(20K) 540 559 1801 801 0.30x 0.67x +NestedLoops(50)x1K 509 501 7023 1887 0.07x 0.27x +Collatz(2K) 185 213 3873 610 0.05x 0.30x ``` -Times in microseconds. WAFER/gf < 1.0 means WAFER is faster. +Times in microseconds. WAFER/gf < 1.0 means WAFER is faster. `sf64` is SwiftForth, +which compiles to native code; two caveats on that column. The install here is an +x86-64 binary under Rosetta 2 while WAFER and gforth are native arm64, so it is a +native-vs-emulated comparison and a native SwiftForth would be faster than these +numbers; and sf64 uses 64-bit cells to WAFER's 32-bit. WAFER is ahead on the four +loop-heavy benchmarks and behind on Fibonacci, which is one call per node with no +loop to promote. ## Remaining Opportunities -| Optimization | Status | Potential Impact | -| -------------------------------- | ------------------- | ----------------------------------------------------- | -| BEGIN loop promotion | Not started | Would speed up GCD-style tight loops further | -| BeginDoubleWhileRepeat promotion | Not started | Rare pattern, low priority | -| LEAVE as IR primitive | Not started | Would enable fast-path for loops with LEAVE | -| Float stack-to-local | Not started | Eliminate float stack memory traffic | -| WASM tail calls proposal | Waiting on wasmtime | Would eliminate stack growth for tail-recursive words | +| Optimization | Status | Potential Impact | +| -------------------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Bounded self-inlining | Not started | Measured 1.33x on Fibonacci, the last benchmark behind sf64. Blocked on `EXIT`: the inliner refuses any body containing one, and a recursive Forth word is `... IF EXIT THEN ... RECURSE`. Needs either a scoped exit (compile an inlined `EXIT` as a branch to the end of a block) or guard-only expansion | +| BeginDoubleWhileRepeat promotion | Not started | Rare pattern, low priority. Its promoted emitter exists but has no loop fixup and is unverified | +| LEAVE as IR primitive | Not started | Would enable fast-path for loops with LEAVE | +| Float stack-to-local | Not started | Eliminate float stack memory traffic | +| WASM tail calls proposal | Waiting on wasmtime | Would eliminate stack growth for tail-recursive words | diff --git a/docs/WAFER.md b/docs/WAFER.md index 4fa93c3..186bae3 100644 --- a/docs/WAFER.md +++ b/docs/WAFER.md @@ -282,11 +282,13 @@ When the compiler encounters a word reference during compilation, it emits: (call_indirect (type $void) (table 0)) ;; indirect call through the table ``` -**Self-recursive optimization**: When a word calls itself (RECURSE), the codegen detects this and emits a direct `call` instead of `call_indirect`, eliminating the table lookup and signature check (~3x faster for recursive words like Fibonacci). +**Self-recursive optimization**: When a word calls itself (RECURSE), the codegen detects this and emits a direct `call` instead of `call_indirect`, eliminating the table lookup and signature check (~3x faster for recursive words like Fibonacci). When the word is also typed, that direct call goes to its fast entry -- see below. **After CONSOLIDATE**: All `call_indirect` between words in the consolidated module are replaced with direct `call` instructions, giving similar benefits for cross-word calls. -At runtime, wasmtime resolves the table entry and calls the target function. Because all functions share the same memory, globals, and table, state passes between words through the data stack in linear memory. There are no function parameters or return values at the WASM level -- everything goes through the stack. +At runtime, wasmtime resolves the table entry and calls the target function. Because all functions share the same memory, globals, and table, state passes between words through the data stack in linear memory. + +**Typed entry points**: that last sentence is the default, not the whole story. A word whose stack effect is statically known also gets a _fast_ entry with signature `(i32 x p) -> (i32 x q)`, which takes its arguments as WASM values and returns its results the same way, so they stay in registers across the call instead of round-tripping through linear memory. The `( -- )` function above is then a wrapper around it, and it is the wrapper that keeps the table slot -- so `EXECUTE`, the outer interpreter, host words and `CATCH` see the memory ABI unchanged. Only a direct call inside the same module takes the fast entry: `RECURSE` in the JIT path, and every resolvable call after `CONSOLIDATE`. See [OPTIMIZATIONS.md](OPTIMIZATIONS.md) section 16. This is subroutine threading: each word is a subroutine, and calling a word is an indirect function call.