Achieve 100% Core compliance, implement CATCH/THROW

Core word set: 0 errors on Gerry Jackson's forth2012-test-suite/core.fr
- Fix POSTPONE for non-immediate words via COMPILE, mechanism
- Fix double-DOES> (WEIRD: pattern) with does-body scanning and
  runtime patching via _DOES_PATCH_
- Implement CATCH/THROW exception handling using wasmtime trap
  mechanism with stack pointer save/restore
- 232 tests passing
This commit is contained in:
2026-03-30 21:26:21 +02:00
parent f15882b518
commit b52b4a79ce
6 changed files with 881 additions and 13 deletions
+4 -1
View File
@@ -23,9 +23,12 @@ jobs:
- uses: Swatinem/rust-cache@v2
- name: Check formatting
- name: Check Rust formatting
run: cargo fmt --all --check
- name: Check Markdown formatting
uses: dprint/check@v2.2
- name: Clippy
run: cargo clippy --workspace -- -D warnings
+8
View File
@@ -1,9 +1,11 @@
# WAFER Project Conventions
## What is WAFER?
WAFER (WebAssembly Forth Engine in Rust) is an optimizing Forth 2012 compiler targeting WebAssembly. Currently a working Forth system with 70+ words and JIT compilation.
## Architecture
- Each Forth word compiles to its own WASM module via `wasm-encoder`
- Modules share memory, globals (dsp/rsp), and a function table via wasmtime imports
- IR-based compilation: Forth -> `Vec<IrOp>` -> WASM codegen -> wasmtime instantiation
@@ -11,6 +13,7 @@ WAFER (WebAssembly Forth Engine in Rust) is an optimizing Forth 2012 compiler ta
- Primitives: either IR-based (compiled to WASM) or host functions (Rust closures in wasmtime)
## Key Files
- `crates/core/src/outer.rs` -- ForthVM: the main runtime, outer interpreter, compiler, all primitives
- `crates/core/src/codegen.rs` -- IR-to-WASM translation, module generation, wasmtime execution tests
- `crates/core/src/dictionary.rs` -- Dictionary data structure with create/find/reveal
@@ -21,11 +24,13 @@ WAFER (WebAssembly Forth Engine in Rust) is an optimizing Forth 2012 compiler ta
## Adding a New Word
**IR primitive** (simple stack/arithmetic/logic -- preferred when possible):
```rust
self.register_primitive("WORD_NAME", false, vec![IrOp::Dup, IrOp::Mul])?;
```
**Host function** (needs Rust logic -- I/O, dictionary manipulation, complex stack access):
```rust
let func = Func::new(&mut self.store, func_type.clone(), move |mut caller, _params, _results| {
// manipulate memory/globals directly
@@ -38,18 +43,21 @@ self.register_host_primitive("WORD_NAME", false, func)?;
Handle in `interpret_token_immediate()` or `compile_token()` as a special case.
## Code Style
- `cargo fmt --all` and `cargo clippy --workspace` must pass with no warnings
- Every public function needs a doc comment
- Use `thiserror` for error types in core crate, `anyhow` for CLI
- Prefer returning `Result` over panicking
## Testing
- Run `cargo test --workspace` before committing (currently 185 tests)
- Forth 2012 compliance: `cargo test -p wafer-core --test compliance`
- Test helper in outer.rs: `eval_output("forth code")` returns printed output as String
- Test helper: `eval_stack("forth code")` returns data stack as Vec<i32>
## Key Principles
1. Correctness first, performance second
2. Maximize Forth, minimize Rust (self-hosting goal -- not yet started)
3. Test-driven: if it's not tested, it doesn't work
+4 -7
View File
@@ -6,7 +6,7 @@ An optimizing Forth 2012 compiler targeting WebAssembly.
## Status
WAFER is a working Forth system. It JIT-compiles each word definition to a separate WASM module and executes via `wasmtime`. 219 unit tests passing, 3 errors on the Forth 2012 Core test suite.
WAFER is a working Forth system. It JIT-compiles each word definition to a separate WASM module and executes via `wasmtime`. 232 unit tests passing, **0 errors on the Forth 2012 Core test suite** (100% Core compliance).
**Working features:**
@@ -117,10 +117,7 @@ tests/ Forth 2012 compliance suite (gerryjackson/forth2012-test-suite sub
### Not Yet Implemented
3 remaining Core test failures:
- `POSTPONE` for non-immediate words in IMMEDIATE context (GT5)
- Double-DOES> in one definition (WEIRD: W1)
- `: NOP : POSTPONE ; ;` meta-programming pattern
All Core words implemented. Exception word set (CATCH/THROW) also available.
## Compliance Status
@@ -128,10 +125,10 @@ Targeting 100% Forth 2012 compliance via [Gerry Jackson's test suite](https://gi
| Word Set | Status |
| ------------------ | ------------------ |
| Core | **97%** (3 failures on test suite) |
| Core | **100%** (0 errors on test suite) |
| Core Extensions | Pending |
| Double-Number | Pending |
| Exception | Pending |
| Exception | **CATCH/THROW implemented** |
| Facility | Pending |
| File-Access | Pending |
| Floating-Point | Pending |
+475 -5
View File
@@ -196,6 +196,12 @@ pub struct ForthVM {
// Pending action from compiled defining/parsing words
// 0 = none, 1 = CONSTANT, 2 = VARIABLE, 3 = CREATE, 4 = EVALUATE
pending_define: Arc<Mutex<i32>>,
// Pending word IDs to compile (used by COMPILE, / POSTPONE mechanism)
pending_compile: Arc<Mutex<Vec<u32>>>,
// Pending DOES> patch: (does_action_id) to apply after word execution
pending_does_patch: Arc<Mutex<Option<u32>>>,
// Exception word set: throw code shared between CATCH and THROW host functions
throw_code: Arc<Mutex<Option<i32>>>,
}
impl ForthVM {
@@ -292,6 +298,9 @@ impl ForthVM {
word_pfa_map: HashMap::new(),
word_pfa_map_shared: None,
pending_define: Arc::new(Mutex::new(0)),
pending_compile: Arc::new(Mutex::new(Vec::new())),
pending_does_patch: Arc::new(Mutex::new(None)),
throw_code: Arc::new(Mutex::new(None)),
};
vm.register_primitives()?;
@@ -655,10 +664,27 @@ impl ForthVM {
return Ok(());
}
"POSTPONE" => {
// Read next token, compile a call to it
// Forth 2012 POSTPONE semantics:
// - Immediate word: compile a call (so it executes at runtime,
// i.e., during compilation of the enclosing definition)
// - Non-immediate word: compile code that, when executed,
// appends Call(word_id) to the current compilation.
// This uses COMPILE, to signal the outer interpreter.
if let Some(next) = self.next_token() {
if let Some((_addr, word_id, _imm)) = self.dictionary.find(&next) {
self.push_ir(IrOp::Call(word_id));
if let Some((_addr, word_id, is_imm)) = self.dictionary.find(&next) {
if is_imm {
// Immediate: just compile a call to it
self.push_ir(IrOp::Call(word_id));
} else {
// Non-immediate: compile code to push xt and call COMPILE,
let compile_comma_id = self
.dictionary
.find("COMPILE,")
.map(|(_, id, _)| id)
.ok_or_else(|| anyhow::anyhow!("POSTPONE: COMPILE, not found"))?;
self.push_ir(IrOp::PushI32(word_id.0 as i32));
self.push_ir(IrOp::Call(compile_comma_id));
}
} else {
anyhow::bail!("POSTPONE: unknown word: {}", next);
}
@@ -716,6 +742,8 @@ impl ForthVM {
if is_immediate {
// Execute immediately even in compile mode
self.execute_word(word_id)?;
// Handle any pending COMPILE, operations from POSTPONE
self.handle_pending_compile();
} else {
self.push_ir(IrOp::Call(word_id));
}
@@ -1098,6 +1126,8 @@ impl ForthVM {
self.sync_base_from_wasm();
// Handle pending defining actions (CONSTANT, VARIABLE, CREATE called at runtime)
self.handle_pending_define()?;
// Handle pending DOES> patch (runtime DOES> from double-DOES> words)
self.handle_pending_does_patch()?;
Ok(())
}
@@ -1395,6 +1425,12 @@ impl ForthVM {
// \ (backslash comment) as an immediate word so POSTPONE can find it
self.register_backslash()?;
// COMPILE, (compile-comma) for POSTPONE mechanism
self.register_compile_comma()?;
// Runtime DOES> patch for double-DOES> support
self.register_does_patch()?;
// CONSTANT, VARIABLE, CREATE as callable words (for use inside colon defs)
self.register_defining_words()?;
@@ -1409,6 +1445,9 @@ impl ForthVM {
// Pictured numeric output
self.register_pictured_numeric()?;
// Exception word set: CATCH and THROW
self.register_catch_throw()?;
Ok(())
}
@@ -2381,6 +2420,120 @@ impl ForthVM {
Ok(())
}
// -----------------------------------------------------------------------
// Exception word set: CATCH and THROW
// -----------------------------------------------------------------------
/// Register CATCH and THROW (Forth 2012 Exception word set).
///
/// CATCH ( xt -- exception# | 0 ) executes xt. If it completes normally,
/// pushes 0. If THROW is called, restores stacks and pushes the throw code.
///
/// THROW ( exception# -- ) if non-zero, unwinds execution back to the
/// nearest CATCH, passing the exception code.
fn register_catch_throw(&mut self) -> anyhow::Result<()> {
let throw_code = Arc::clone(&self.throw_code);
let memory = self.memory;
let dsp = self.dsp;
let rsp = self.rsp;
let table = self.table;
// THROW ( exception# -- )
let throw_code_for_throw = Arc::clone(&throw_code);
let throw_func = Func::new(
&mut self.store,
FuncType::new(&self.engine, [], []),
move |mut caller, _params, _results| {
// Pop throw code from data stack
let sp = dsp.get(&mut caller).unwrap_i32() as u32;
if sp >= DATA_STACK_TOP {
return Err(wasmtime::Error::msg("THROW: stack underflow"));
}
let data = memory.data(&caller);
let b: [u8; 4] = data[sp as usize..sp as usize + 4].try_into().unwrap();
let code = i32::from_le_bytes(b);
// Pop TOS
dsp.set(&mut caller, Val::I32((sp + CELL_SIZE) as i32))?;
if code == 0 {
return Ok(());
}
// Store the throw code and trigger a trap to unwind back to CATCH
*throw_code_for_throw.lock().unwrap() = Some(code);
Err(wasmtime::Error::msg("forth-throw"))
},
);
self.register_host_primitive("THROW", false, throw_func)?;
// CATCH ( xt -- exception# | 0 )
let throw_code_for_catch = Arc::clone(&throw_code);
let catch_func = Func::new(
&mut self.store,
FuncType::new(&self.engine, [], []),
move |mut caller, _params, _results| {
// Pop xt from data stack
let sp = dsp.get(&mut caller).unwrap_i32() as u32;
if sp >= DATA_STACK_TOP {
return Err(wasmtime::Error::msg("CATCH: stack underflow"));
}
let data = memory.data(&caller);
let b: [u8; 4] = data[sp as usize..sp as usize + 4].try_into().unwrap();
let xt = u32::from_le_bytes(b);
// Pop TOS (remove xt)
let sp_after_pop = sp + CELL_SIZE;
dsp.set(&mut caller, Val::I32(sp_after_pop as i32))?;
// Save stack depths for restoration on THROW
let saved_dsp = sp_after_pop;
let saved_rsp = rsp.get(&mut caller).unwrap_i32() as u32;
// Look up the function in the table
let func_ref = table
.get(&mut caller, xt as u64)
.ok_or_else(|| wasmtime::Error::msg("CATCH: invalid xt"))?;
let func = *func_ref
.unwrap_func()
.ok_or_else(|| wasmtime::Error::msg("CATCH: null funcref"))?;
// Call the word -- if THROW is invoked, func.call returns Err
match func.call(&mut caller, &[], &mut []) {
Ok(()) => {
// Normal completion: push 0
let current_sp = dsp.get(&mut caller).unwrap_i32() as u32;
let new_sp = current_sp.wrapping_sub(CELL_SIZE);
let data = memory.data_mut(&mut caller);
data[new_sp as usize..new_sp as usize + 4]
.copy_from_slice(&0_i32.to_le_bytes());
dsp.set(&mut caller, Val::I32(new_sp as i32))?;
Ok(())
}
Err(_) => {
// Check if this was a THROW (vs some other trap)
let mut tc = throw_code_for_catch.lock().unwrap();
let code = tc.take().unwrap_or(-1);
drop(tc);
// Restore stack pointers to saved depths
dsp.set(&mut caller, Val::I32(saved_dsp as i32))?;
rsp.set(&mut caller, Val::I32(saved_rsp as i32))?;
// Push the throw code onto the restored stack
let new_sp = saved_dsp.wrapping_sub(CELL_SIZE);
let data = memory.data_mut(&mut caller);
data[new_sp as usize..new_sp as usize + 4]
.copy_from_slice(&code.to_le_bytes());
dsp.set(&mut caller, Val::I32(new_sp as i32))?;
Ok(())
}
}
},
);
self.register_host_primitive("CATCH", false, catch_func)?;
Ok(())
}
// -----------------------------------------------------------------------
// EVALUATE -- save input, interpret string, restore input
// -----------------------------------------------------------------------
@@ -2570,7 +2723,61 @@ impl ForthVM {
does_tokens.push(token);
}
// Compile the does-body as a separate word
// Check for a second DOES> in the does-body (double-DOES> pattern).
// If found, split: first part is the first does-action, second part
// becomes a separate does-action that gets patched in at runtime.
let does_split = does_tokens
.iter()
.position(|t| t.eq_ignore_ascii_case("DOES>"));
let (first_tokens, second_does_tokens) = if let Some(pos) = does_split {
(
does_tokens[..pos].to_vec(),
Some(does_tokens[pos + 1..].to_vec()),
)
} else {
(does_tokens, None)
};
// If there's a second DOES>, compile its body first as a separate word
let second_does_action_id = if let Some(ref second_tokens) = second_does_tokens {
let second_word_id = self
.dictionary
.create("_does_action2_", false)
.map_err(|e| anyhow::anyhow!("{}", e))?;
self.dictionary.reveal();
self.next_table_index = self.next_table_index.max(second_word_id.0 + 1);
let saved_name2 = self.compiling_name.take();
let saved_word_id2 = self.compiling_word_id.take();
let saved_control2 = std::mem::take(&mut self.control_stack);
self.compiling_ir.clear();
self.compiling_name = Some("_does_action2_".to_string());
self.compiling_word_id = Some(second_word_id);
for token in second_tokens {
self.compile_token(token)?;
}
let second_ir = std::mem::take(&mut self.compiling_ir);
let config = CodegenConfig {
base_fn_index: second_word_id.0,
table_size: self.table_size(),
};
let compiled = compile_word("_does_action2_", &second_ir, &config)
.map_err(|e| anyhow::anyhow!("codegen error for DOES> body 2: {}", e))?;
self.instantiate_and_install(&compiled, second_word_id)?;
self.compiling_name = saved_name2;
self.compiling_word_id = saved_word_id2;
self.control_stack = saved_control2;
Some(second_word_id)
} else {
None
};
// Compile the first does-body as a separate word
let does_word_id = self
.dictionary
.create("_does_action_", false)
@@ -2587,10 +2794,21 @@ impl ForthVM {
self.compiling_name = Some("_does_action_".to_string());
self.compiling_word_id = Some(does_word_id);
for token in &does_tokens {
for token in &first_tokens {
self.compile_token(token)?;
}
// If there's a second DOES>, append code to patch the word at runtime
if let Some(second_action_id) = second_does_action_id {
let does_patch_id = self
.dictionary
.find("_DOES_PATCH_")
.map(|(_, id, _)| id)
.ok_or_else(|| anyhow::anyhow!("_DOES_PATCH_ not found"))?;
self.push_ir(IrOp::PushI32(second_action_id.0 as i32));
self.push_ir(IrOp::Call(does_patch_id));
}
let does_ir = std::mem::take(&mut self.compiling_ir);
let config = CodegenConfig {
base_fn_index: does_word_id.0,
@@ -3295,6 +3513,64 @@ impl ForthVM {
// CONSTANT, VARIABLE, CREATE as callable defining words
// -----------------------------------------------------------------------
/// Register COMPILE, as a host function.
/// COMPILE, ( xt -- ) appends a call to xt into the current compilation.
/// Used internally by POSTPONE for non-immediate words.
fn register_compile_comma(&mut self) -> anyhow::Result<()> {
let memory = self.memory;
let dsp = self.dsp;
let pending_compile = Arc::clone(&self.pending_compile);
let func = Func::new(
&mut self.store,
FuncType::new(&self.engine, [], []),
move |mut caller, _params, _results| {
// Pop xt from data stack
let sp = dsp.get(&mut caller).unwrap_i32() as u32;
let data = memory.data(&caller);
let b: [u8; 4] = data[sp as usize..sp as usize + 4].try_into().unwrap();
let xt = u32::from_le_bytes(b);
// Drop top of stack
let new_sp = sp + 4;
dsp.set(&mut caller, Val::I32(new_sp as i32)).unwrap();
// Signal the outer interpreter to compile a call to this xt
pending_compile.lock().unwrap().push(xt);
Ok(())
},
);
self.register_host_primitive("COMPILE,", false, func)?;
Ok(())
}
/// Register `_does_patch_` as a host function for runtime DOES> patching.
/// ( does_action_id -- ) Signals the outer interpreter to patch the most
/// recently CREATEd word with a new DOES> action.
fn register_does_patch(&mut self) -> anyhow::Result<()> {
let memory = self.memory;
let dsp = self.dsp;
let pending_does_patch = Arc::clone(&self.pending_does_patch);
let func = Func::new(
&mut self.store,
FuncType::new(&self.engine, [], []),
move |mut caller, _params, _results| {
// Pop does_action_id from data stack
let sp = dsp.get(&mut caller).unwrap_i32() as u32;
let data = memory.data(&caller);
let b: [u8; 4] = data[sp as usize..sp as usize + 4].try_into().unwrap();
let does_action_id = u32::from_le_bytes(b);
let new_sp = sp + 4;
dsp.set(&mut caller, Val::I32(new_sp as i32)).unwrap();
*pending_does_patch.lock().unwrap() = Some(does_action_id);
Ok(())
},
);
self.register_host_primitive("_DOES_PATCH_", false, func)?;
Ok(())
}
/// Register CONSTANT, VARIABLE, CREATE as host functions so they can
/// be compiled into colon definitions (e.g., `: EQU CONSTANT ;`).
fn register_defining_words(&mut self) -> anyhow::Result<()> {
@@ -3417,6 +3693,54 @@ impl ForthVM {
}
}
/// Drain pending_compile and push IrOp::Call for each entry into compiling_ir.
/// Called after executing an immediate word during compilation.
fn handle_pending_compile(&mut self) {
let pending: Vec<u32> = {
let mut v = self.pending_compile.lock().unwrap();
std::mem::take(&mut *v)
};
for xt in pending {
self.push_ir(IrOp::Call(WordId(xt)));
}
}
/// Handle a pending runtime DOES> patch.
/// When a DOES> body contains another DOES>, the inner DOES> signals via
/// `_DOES_PATCH_` to replace the most recently CREATEd word's behavior.
fn handle_pending_does_patch(&mut self) -> anyhow::Result<()> {
let does_action_id = {
let mut p = self.pending_does_patch.lock().unwrap();
p.take()
};
if let Some(action_id) = does_action_id {
let (target_addr, pfa) = self
.last_created_info
.ok_or_else(|| anyhow::anyhow!("runtime DOES>: no CREATEd word to patch"))?;
let fn_index = self
.dictionary
.code_field(target_addr)
.map_err(|e| anyhow::anyhow!("{}", e))?;
let target_word_id = WordId(fn_index);
let name = self
.dictionary
.word_name(target_addr)
.map_err(|e| anyhow::anyhow!("{}", e))?;
let patched_ir = vec![IrOp::PushI32(pfa as i32), IrOp::Call(WordId(action_id))];
let config = CodegenConfig {
base_fn_index: target_word_id.0,
table_size: self.table_size(),
};
let compiled = compile_word(&name, &patched_ir, &config)
.map_err(|e| anyhow::anyhow!("runtime DOES> patch codegen: {}", e))?;
self.instantiate_and_install(&compiled, target_word_id)?;
}
Ok(())
}
// -----------------------------------------------------------------------
// Backslash comment as a compilable immediate word
// -----------------------------------------------------------------------
@@ -4759,4 +5083,150 @@ mod tests {
let len = data[addr as usize];
assert_eq!(len, 5); // "HELLO" is 5 chars
}
// ===================================================================
// Exception word set: CATCH and THROW
// ===================================================================
#[test]
fn test_catch_no_throw() {
// CATCH with a word that doesn't throw should push 0
assert_eq!(eval_output(": TEST ['] DUP CATCH . ; 5 TEST"), "0 ");
}
#[test]
fn test_catch_no_throw_stack() {
// After CATCH of a non-throwing word, TOS should be 0 and the
// word's effect should be visible underneath
assert_eq!(eval_stack("5 ' DUP CATCH"), vec![0, 5, 5]);
}
#[test]
fn test_throw_zero_is_noop() {
// THROW with 0 should do nothing
assert_eq!(eval_output(": TEST 0 THROW 123 . ; TEST"), "123 ");
}
#[test]
fn test_catch_throw_basic() {
// CATCH with a word that throws should push the throw code
assert_eq!(
eval_output(": THROWER 42 THROW ; : TEST ['] THROWER CATCH . ; TEST"),
"42 "
);
}
#[test]
fn test_catch_stack_restore() {
// THROW should restore the data stack to the depth saved by CATCH
// Before CATCH: stack is (10 20), CATCH pops xt, saves depth (10 20)
// THROWER pushes 1 2 3 then throws 99
// CATCH restores to (10 20) and pushes 99
let stack = eval_stack(": THROWER 1 2 3 99 THROW ; 10 20 ' THROWER CATCH");
assert_eq!(stack, vec![99, 20, 10]);
}
#[test]
fn test_nested_catch() {
// Nested CATCH: inner CATCH catches the throw, outer CATCH sees success
assert_eq!(
eval_output(
": INNER 5 THROW ; : OUTER ['] INNER CATCH . ; : TEST ['] OUTER CATCH . ; TEST"
),
"5 0 "
);
}
#[test]
fn test_catch_negative_throw() {
// Standard throw codes are negative
assert_eq!(
eval_output(": THROWER -1 THROW ; : TEST ['] THROWER CATCH . ; TEST"),
"-1 "
);
}
#[test]
fn test_catch_preserves_output() {
// Output before THROW should still be visible
assert_eq!(
eval_output(": THROWER 65 EMIT 1 THROW ; : TEST ['] THROWER CATCH DROP ; TEST"),
"A"
);
}
#[test]
fn test_catch_in_colon_def() {
// CATCH can be used inside a colon definition
assert_eq!(
eval_output(": ERR 10 THROW ; : SAFE ['] ERR CATCH ; SAFE ."),
"10 "
);
}
#[test]
fn test_throw_skips_rest_of_word() {
// After THROW, remaining code in the throwing word should not execute
assert_eq!(
eval_output(": BAD 1 THROW 999 . ; : TEST ['] BAD CATCH . ; TEST"),
"1 "
);
}
// ===================================================================
// POSTPONE: Forth 2012 GT5/GT7 tests
// ===================================================================
#[test]
fn test_postpone_non_immediate_gt5() {
// : GT1 123 ;
// : GT4 POSTPONE GT1 ; IMMEDIATE
// : GT5 GT4 ;
// GT5 -> 123
let mut vm = ForthVM::new().unwrap();
vm.evaluate(": GT1 123 ;").unwrap();
vm.evaluate(": GT4 POSTPONE GT1 ; IMMEDIATE").unwrap();
vm.evaluate(": GT5 GT4 ;").unwrap();
vm.evaluate("GT5").unwrap();
assert_eq!(vm.data_stack(), vec![123]);
}
#[test]
fn test_postpone_immediate_gt7() {
// : GT6 345 ; IMMEDIATE
// : GT7 POSTPONE GT6 ;
// GT7 -> 345
let mut vm = ForthVM::new().unwrap();
vm.evaluate(": GT6 345 ; IMMEDIATE").unwrap();
vm.evaluate(": GT7 POSTPONE GT6 ;").unwrap();
vm.evaluate("GT7").unwrap();
assert_eq!(vm.data_stack(), vec![345]);
}
// ===================================================================
// Double DOES>: Forth 2012 WEIRD: W1 test
// ===================================================================
#[test]
fn test_double_does() {
// : WEIRD: CREATE DOES> 1 + DOES> 2 + ;
// WEIRD: W1
// W1 first call: PFA 1 + (first DOES> behavior, then patches to second)
// W1 second call: PFA 2 + (second DOES> behavior)
let mut vm = ForthVM::new().unwrap();
vm.evaluate(": WEIRD: CREATE DOES> 1 + DOES> 2 + ;")
.unwrap();
vm.evaluate("WEIRD: W1").unwrap();
// Get HERE (which is the PFA of W1)
vm.evaluate("' W1 >BODY").unwrap();
let pfa = vm.data_stack()[0];
vm.evaluate("DROP").unwrap();
// First call: PFA 1 +
vm.evaluate("W1").unwrap();
assert_eq!(vm.data_stack(), vec![pfa + 1]);
vm.evaluate("DROP").unwrap();
// Second call: PFA 2 +
vm.evaluate("W1").unwrap();
assert_eq!(vm.data_stack(), vec![pfa + 2]);
}
}
+376
View File
@@ -0,0 +1,376 @@
# How WAFER Works
WAFER (WebAssembly Forth Engine in Rust) is a Forth 2012 compiler that JIT-compiles each Forth word to its own WebAssembly module and executes it via [wasmtime](https://wasmtime.dev/). This document describes concretely what happens at each step -- from startup through word definition to execution. For background on the Forth language itself, see [FORTH.md](FORTH.md).
## Project Layout
```
crates/
core/src/
outer.rs ForthVM: outer interpreter, compiler, all primitives
codegen.rs IR-to-WASM translation, module generation
dictionary.rs Dictionary (linked list in Vec<u8>)
ir.rs IrOp enum -- the intermediate representation
memory.rs Memory layout constants (addresses, sizes)
error.rs Error types
cli/src/
main.rs CLI REPL (rustyline), file execution
web/src/
lib.rs Browser bindings (planned)
forth/
prelude.fth Standard library loader (planned)
tests/
forth2012-test-suite/ Forth 2012 compliance test suite (submodule)
```
The entire compiler and runtime lives in `outer.rs` (~5200 lines). Codegen is in `codegen.rs` (~1500 lines). Everything else is supporting infrastructure.
## What Happens When You Start WAFER
Running `cargo run -p wafer` or the `wafer` binary triggers this sequence:
### 1. ForthVM::new()
`ForthVM::new()` in `outer.rs` creates the shared WebAssembly infrastructure:
```
wasmtime Engine Compilation engine (shared across all modules)
wasmtime Store Runtime state container
Linear Memory 16 pages (1 MiB), expandable to 256 pages (16 MiB)
Global: DSP Data stack pointer, initialized to 0x1540
Global: RSP Return stack pointer, initialized to 0x2540
Function Table 256 funcref entries (grows as needed)
```
A host function is created in Rust and made available as a WASM import:
- **emit** -- takes an `i32` character code, appends it to the output buffer
This is the only host function that compiled WASM modules import directly. Other I/O words like `.` (dot) are registered as host primitives that live in the function table but are implemented entirely in Rust.
### 2. register_primitives()
For each of the ~80 built-in words (DUP, DROP, +, -, @, !, IF, DO, EMIT, ...), the system:
1. Creates a HIDDEN entry in the dictionary
2. Compiles the word's IR body to a WASM module via `compile_word()`
3. Instantiates the module with wasmtime, linking it to the shared memory, globals, and table
4. Installs the compiled function in the function table at the word's index
5. Reveals the dictionary entry (removes HIDDEN flag)
After this, the dictionary contains all built-in words and the function table has a compiled WASM function for each one.
### 3. REPL loop
The CLI enters a `rustyline` loop. The prompt is `> ` in interpret mode and ` ] ` in compile mode. Each line is passed to `vm.evaluate()`, which feeds it to the outer interpreter.
## Memory Layout
All stacks and data structures live in a single WASM linear memory. Addresses are fixed at compile time in `memory.rs`:
```
Address Region Size Notes
------- ------ ---- -----
0x0000 System variables 64 B STATE, BASE, >IN, HERE, LATEST, HLD, ...
0x0040 Input buffer 1024 B Current source line being parsed
0x0440 PAD 256 B Scratch area for string formatting
0x0540 Data stack 4096 B 1024 cells, grows downward
0x1540 Return stack 4096 B 1024 cells, grows downward
0x2540 Float stack 2048 B 256 doubles, grows downward
0x2D40 Dictionary variable Linked list of word headers, grows upward
...
0x10000 User data space variable VARIABLE, CREATE, ALLOT data, grows upward
```
The data stack pointer (DSP) starts at 0x1540 (the top of the data stack region) and decrements by 4 bytes for each push. The return stack pointer (RSP) starts at 0x2540 and works the same way.
## What Happens When You Type `5 3 + .`
The outer interpreter in `evaluate()` tokenizes the input by whitespace and processes each token:
### Token: `5`
1. `dictionary.find("5")` -- not found
2. `parse_number("5")` -- succeeds, returns 5
3. `push_data_stack(5)`:
- Read DSP global (0x1540)
- Decrement by 4 → 0x153C
- Write `5` to memory at 0x153C
- Update DSP to 0x153C
### Token: `3`
Same path. Stack is now:
```
0x1538: 3 ← DSP (top of stack)
0x153C: 5
```
### Token: `+`
1. `dictionary.find("+")` -- found, returns WordId and function table index
2. `execute_word(word_id)`:
- Look up function reference in `table[word_id]`
- Call it via wasmtime: `func.call(&mut store, &[], &mut [])`
3. The compiled WASM function for `+` executes:
- Load value at DSP (3), increment DSP
- Load value at DSP (5), increment DSP
- Compute 5 + 3 = 8
- Decrement DSP, store 8
Stack is now: `0x153C: 8 ← DSP`
### Token: `.`
1. `dictionary.find(".")` -- found (host function primitive)
2. `execute_word(word_id)` -- calls the Rust `dot` closure:
- Reads value at DSP (8), increments DSP
- Formats as `"8 "` and appends to the output buffer
3. Back in the REPL, output buffer is printed: `8 ok`
## What Happens When You Define a Word
### `: square dup * ;`
#### Token: `:`
The colon handler in `interpret_token()`:
1. Reads the next token from input: `"square"`
2. `dictionary.create("square", false)` -- creates a new HIDDEN entry, returns `WordId(N)` where N is the next function table index
3. Clears `compiling_ir` (the IR accumulator)
4. Sets `state = -1` (compile mode)
5. Prompt changes to ` ] `
#### Token: `dup` (in compile mode)
1. `dictionary.find("dup")` -- found, not IMMEDIATE
2. Since we are compiling and the word is not immediate, append `IrOp::Call(dup_word_id)` to `compiling_ir`
#### Token: `*` (in compile mode)
1. `dictionary.find("*")` -- found, not IMMEDIATE
2. Append `IrOp::Call(mul_word_id)` to `compiling_ir`
#### Token: `;`
The semicolon handler triggers `finish_colon_def()`:
**Step 1 -- Take the IR:**
```rust
compiling_ir = [IrOp::Call(WordId(0)), // DUP
IrOp::Call(WordId(6))] // *
```
(Actual WordId values depend on registration order.)
**Step 2 -- Codegen:**
`compile_word("square", &ir, &config)` in `codegen.rs` generates a complete WASM module:
```wasm
;; Type section
(type $void (func))
(type $i32 (func (param i32)))
;; Import section -- shared resources
(import "env" "emit" (func $emit (type $i32)))
(import "env" "memory" (memory 16))
(import "env" "dsp" (global $dsp (mut i32)))
(import "env" "rsp" (global $rsp (mut i32)))
(import "env" "table" (table 256 funcref))
;; Function section
(func $fn (type $void)
;; Call DUP via function table
(i32.const 0) ;; DUP's table index
(call_indirect (type $void) (table 0))
;; Call * via function table
(i32.const 6) ;; MUL's table index
(call_indirect (type $void) (table 0))
)
;; Export section
(export "fn" (func $fn))
;; Element section -- install in table
(elem (i32.const N) func $fn) ;; N = square's WordId
```
This module is generated as raw WASM bytecode using `wasm-encoder`, not as text. The pseudocode above is for illustration.
**Step 3 -- Instantiate and install:**
```
Module::new(&engine, &wasm_bytes) Parse WASM
Instance::new(&store, &module, &[ Link to shared resources:
emit_func, - emit host function
memory, - shared linear memory
dsp, - data stack pointer global
rsp, - return stack pointer global
table, - shared function table
])
table.set(N, exported_fn) Install in function table
```
**Step 4 -- Reveal:**
`dictionary.reveal()` removes the HIDDEN flag from "square". The word is now visible to `FIND` and can be called.
**Step 5 -- Return to interpret mode:**
`state = 0`, prompt returns to `> `.
### What happens when you then type `7 square .`
- `7` -- pushed to data stack (same as before)
- `square` -- `dictionary.find("SQUARE")` returns WordId(N), `execute_word(N)` calls the compiled function, which in turn calls DUP and * via `call_indirect`, leaving 49 on the stack
- `.` -- pops 49, prints `49 `
Output: `49 ok`
## The Compilation Pipeline
```
Forth source
Outer Interpreter (outer.rs)
Tokenizes by whitespace, looks up each token in dictionary.
In interpret mode: executes words, pushes numbers.
In compile mode: builds Vec<IrOp>.
IR (ir.rs)
~50 operation types: PushI32, Dup, Add, Call(WordId),
If { then_body, else_body }, DoLoop { body }, Exit, ...
Control flow is nested: If/DoLoop/BeginUntil contain Vec<IrOp> bodies.
Codegen (codegen.rs)
compile_word() walks the IR and emits WASM instructions via wasm-encoder.
Each word becomes a standalone WASM module that imports the shared resources.
The function is exported as "fn" and placed in the table via the element section.
Wasmtime
Module::new() parses and validates the WASM.
Instance::new() links imports and instantiates.
The exported function is extracted and stored in the function table.
Function Table
Every word has a unique index. Calling a word = call_indirect with that index.
All modules share the same table, memory, and globals.
```
## How Words Call Each Other
Every word -- primitive or user-defined -- gets a unique `WordId`, which is its index in the shared function table.
When the compiler encounters a word reference during compilation, it emits:
```wasm
(i32.const <word_id>) ;; push the function table index
(call_indirect (type $void) (table 0)) ;; indirect call through the table
```
At runtime, wasmtime resolves the table entry and calls the target function. Because all functions share the same memory, globals, and table, state passes between words through the data stack in linear memory. There are no function parameters or return values at the WASM level -- everything goes through the stack.
This is subroutine threading: each word is a subroutine, and calling a word is an indirect function call.
## Dictionary Structure
The dictionary is a linked list stored in a `Vec<u8>` buffer on the Rust side (in the `Dictionary` struct). Each entry has this layout:
```
Offset Size Contents
------ ---- --------
+0 4 Link: address of previous dictionary entry (0 = end of list)
+4 1 Flags: IMMEDIATE (0x80) | HIDDEN (0x40) | name_length (0-31)
+5 N Name: N bytes, stored in uppercase
+5+N 0-3 Padding to 4-byte alignment
4 Code field: function table index (WordId)
... Parameter field: used by VARIABLE, CONSTANT, DOES>, etc.
```
**FIND** walks the list backward from `LATEST` (the most recent entry). For each entry:
1. Skip if HIDDEN flag is set
2. Compare name length, then bytes (case-insensitive)
3. If match: return the entry address, WordId (from code field), and IMMEDIATE flag
4. Otherwise follow the link to the previous entry
This means later definitions shadow earlier ones, and words being compiled (HIDDEN) are invisible to FIND until `;` reveals them.
## Primitives: Two Kinds
### IR Primitives
Most built-in words are defined as IR sequences and compiled to WASM like any user-defined word:
```rust
self.register_primitive("DUP", false, vec![IrOp::Dup])?;
self.register_primitive("+", false, vec![IrOp::Add])?;
self.register_primitive("OVER", false, vec![IrOp::Over])?;
self.register_primitive("ABS", false, vec![IrOp::Dup, IrOp::ZeroLt,
IrOp::If { then_body: vec![IrOp::Negate], else_body: None }])?;
```
The codegen translates each `IrOp` to inline WASM instructions. For example, `IrOp::Dup` becomes:
```wasm
;; peek: read top-of-stack without popping
(global.get $dsp)
(i32.load) ;; TOS value now on WASM operand stack
;; push_via_local: store value back to a new stack slot
(local.set 0) ;; save to scratch local
(global.get $dsp) ;; dsp -= 4 (grow stack)
(i32.const 4)
(i32.sub)
(global.set $dsp)
(global.get $dsp) ;; mem[dsp] = saved value
(local.get 0)
(i32.store)
```
### Host Primitives
Words that need access to Rust-side state (output buffer, dictionary, user variables) are implemented as Rust closures and placed directly in the function table:
```rust
let func = Func::new(&mut store, void_type, move |mut caller, _, _| {
// Read/write memory and globals via wasmtime's Caller API
Ok(())
});
self.register_host_primitive(".", false, func)?;
```
Examples: `.` (print number), `HERE` (return user data pointer), `WORDS` (list dictionary), `TYPE` (print string from memory).
Note: `EMIT` is an IR primitive -- it compiles to WASM code that calls the imported `emit` host function. This is different from host primitives like `.`, which are Rust closures placed directly in the function table.
## What WAFER Does NOT Create on Disk
WAFER generates all WASM modules in memory. No `.wasm` files are written to disk. No caches, no configuration files, no persistent state. Every time you start WAFER, it rebuilds everything from scratch.
The `--consolidate` CLI flag is reserved for a planned feature: compiling all words into a single optimized WASM module for ahead-of-time deployment. This is not yet implemented.
## Running the Tests
```bash
cargo test --workspace # All unit tests (~220)
cargo test -p wafer-core --test compliance # Forth 2012 compliance suite
cargo run -p wafer -- file.fth # Execute a Forth source file
echo '5 3 + .' | cargo run -p wafer # Pipe input (non-interactive)
```
Test helpers in `outer.rs` for writing Rust-side tests:
```rust
assert_eq!(eval_output("5 3 + ."), "8 ");
assert_eq!(eval_stack("1 2 3"), vec![1, 2, 3]);
```
+14
View File
@@ -0,0 +1,14 @@
{
"markdown": {
"lineWidth": 120,
"textWrap": "maintain"
},
"includes": ["**/*.md"],
"excludes": [
"target/",
"tests/forth2012-test-suite/"
],
"plugins": [
"https://plugins.dprint.dev/markdown-0.21.1.wasm"
]
}