12 Commits

Author SHA1 Message Date
Oleksandr Kozachuk 972c9544e4 chore(tools): sync bat syntax with current word set
Alternations diffed against live WORDS output (304 words) plus
outer-interpreter tokens. Adds float transcendentals, double-cell
ops, pictured numeric, conditional compilation, string ops,
SEE/SEE-IR/DUMP/BYE/HELP, RP@/RDEPTH/.RS, REMEMBER/EMPTY/GILD,
DECIMAL/HEX, INCLUDE/INCLUDED, WITHIN, DEFER!/DEFER@, C,.
2026-08-06 12:23:20 +02:00
Oleksandr Kozachuk 5d40f32953 build: add just install recipe
Installs bat Forth syntax, then cargo install --locked the CLI.
CARGO_PROFILE_RELEASE_STRIP=none because Cargo's release default
(strip = "debuginfo") emits dylibs macOS 27 dyld rejects with
"mis-aligned LINKEDIT string pool"; proc macros then fail to load
during the build.
2026-08-06 12:23:02 +02:00
Oleksandr Kozachuk 4ffa67e784 feat(core): INCLUDE, error overhaul, sf64 lane, WORDS ALL, .RS
WS-012 -- INCLUDE/INCLUDED:
- Injected source loader (core stays IO-free: CLI installs a
  filesystem reader, web leaves it unset -> defined error). Recursive
  include_file feeds files line-by-line through evaluate, so compile
  state and SEE capture span lines for free. Cycle detection, depth
  cap 16, paths relative to the including file, SOURCE-ID per nesting
  level, parent input restored on success/error/BYE.
- CLI file mode now runs through the include machinery: `wafer x.fth`
  gets file:line error context and a base dir for nested INCLUDEs.
- Unlocks the REMEMBER+INCLUDE reload loop.

WS-008 -- error reporting remainder:
- Errors inside included files carry `file.fth:12:` context
  (anyhow context chain; CLI prints {e:#}).
- describe_uncaught now returns typed WaferError::UncaughtThrow
  { code, message } -- display text unchanged, THROW code reachable
  via downcast for CLI/web consumers.
- compile_word emits a WASM name section; wasmtime trap backtraces
  name the faulting word and runtime_native prefixes "in <WORD>:".
  Batch/consolidated modules stay unnamed (no name plumbing there;
  boot primitives rarely trap).

WS-003 -- SwiftForth correctness lane:
- compare_all_programs_sf64 runs the program corpus with sf64 as
  oracle; whitespace-token comparison (sf64 prints numbers
  space-prefixed and echoes piped lines). 34/35 parity; dot-quote
  skipped (interpret-mode ." is a SwiftForth no-op). #[ignore]d like
  the gforth lane; `just compare-correctness` runs both.

WS-011 leftovers:
- WORDS ALL: grouped full view -- one section per wordlist (search
  order first), then internal words, each with counts. Backed by
  Dictionary::visible_entries (name, wid, internal); visible_words
  now derives from it.
- .RS / RDEPTH: return-stack introspection in boot.fth over a new
  RP@ primitive (IrOp::RpFetch); BEGIN/WHILE walk so the walk never
  touches the stack it prints. SPACES clamped per 6.1.2230.

549 unit + 11 compliance + 9(+2) comparison + 5 crypto + 1 bench
green; fmt/clippy clean; --no-default-features and wasm32 web checks
pass.
2026-08-06 12:03:29 +02:00
Oleksandr Kozachuk 380250a641 feat(core): SEE, SEE-IR, HELP introspection trio (WS-010)
Implements plans/01-see-introspection.md, all phases.

- see.rs: feature-free IR pretty-printer (format_ir/format_ir_with),
  exhaustive over IrOp -- a new variant fails the build, not the output.
- SEE-IR <name>: post-optimization IR view with resolved callee names,
  immediate/does> annotations; host-word and interpreter-token stubs.
- SEE <name>: verbatim source capture for colon words (multi-line,
  comments preserved, EVALUATE-nesting safe, error-path wiped, MARKER/
  REMEMBER/EMPTY roll word sources back too). Data definers (VARIABLE/
  CONSTANT/CREATE/BUFFER:/2*/F*/SYNONYM) record synthesized one-liners
  at definition time; VALUE/2VALUE/FVALUE/DEFER synthesize at SEE time
  so current values and IS targets show. Fallback chain ends at IR dump
  or host-word stub -- SEE never dead-ends on a defined word.
- HELP [<name>]: wordhelp.rs doc table with stack effect + one-line
  description for EVERY word in a fresh VM (300+ dictionary words plus
  all outer-interpreter tokens); a coverage test fails the build if a
  word is ever added undocumented. User words echo their leading
  ( ... -- ... ) comment. SEE/SEE-IR prepend the HELP line as a
  \ comment. Bare HELP prints usage.
- boot.fth colon definitions get real sources for free (they flow
  through evaluate); INTERPRETER_TOKENS gained the missing ?DO.

524 unit + 11 compliance + 9 comparison + 5 crypto + 1 bench green;
fmt/clippy clean; core still builds --no-default-features; web
wasm-pack build unchanged.
2026-08-06 11:18:03 +02:00
Oleksandr Kozachuk cda296aab5 feat(codegen): stack under/overflow guards in compiled words (WS-007)
Compiled code could silently move dsp/rsp/fsp out of their stack
regions (e.g. DROP on an empty stack), corrupting later pushes with
no diagnostic -- the addresses stay inside valid linear memory, so
nothing could trap. Host-side checks cannot catch it.

- Guards are emitted at the sp-adjustment choke points (dsp_inc/
  dsp_dec, fsp_inc/fsp_dec, rpush/rpop/rpeek, peek, TwoDup/TwoDrop,
  promoted prologue/epilogue -- DROP never loads its value, so
  guarding pop() alone is not enough). On fault: write the code to
  SYSVAR_FAULT_CODE, call _STACK_FAULT_, which THROWs it -- so
  guards are CATCHable and print standard messages (-3/-4/-5/-6/
  -44/-45).
- The batch/consolidated compile path (all boot primitives) and the
  export path are wired too; a thread-local carries the fault index
  into the shared emission helpers.
- Config: codegen.stack_guards, default ON. `wafer build` output
  defaults OFF (production artifact); WAFER_STACK_GUARDS=0|1
  overrides either. Perf comparison lanes run unguarded.
- Measured overhead in release loops: within noise (never-taken
  branches).
- toolstest.fth baseline 37 -> 38: line 368's bare interpreted `R>`
  used to underflow silently and count as passing; the guard now
  correctly reports -6.
2026-08-05 17:00:22 +02:00
Oleksandr Kozachuk e31407ab58 feat(core): REMEMBER + EMPTY/GILD; marker rollback covers namespace state
- REMEMBER <name>: SwiftForth-style re-runnable marker -- restores
  to just AFTER its own definition and survives execution. The
  edit-reload-test loop: REMEMBER fresh ... fresh ... fresh.
- EMPTY rolls back to the boot dictionary; GILD re-baselines EMPTY
  to the current state. Baseline captured at VM construction.
- MarkerState now also snapshots search order, wid allocation,
  compilation wordlist, REPLACES table, and ABORT" texts; restore
  discards marker entries newer than the snapshot (was: newer than
  the executing marker id only).
- Shared snapshot_marker_state/apply_marker_state used by MARKER,
  REMEMBER, EMPTY, and GILD.

Known ambiguity (standard-conformant): a DEFER defined before a
marker but retargeted at a word defined after it dangles after
rollback; stale function-table slots are name-unreachable and get
overwritten by later definitions.
2026-08-05 16:22:47 +02:00
Oleksandr Kozachuk 2910884b83 fix(repl): multi-line output starts on its own line, inline ok only for single-line 2026-08-05 16:07:10 +02:00
Oleksandr Kozachuk f584066a0a feat(repl): usability batch - errors, WORDS, .S, BYE, DUMP, history
Core:
- Uncaught THROW prints its standard message ("Stack underflow
  (throw -4)"; unknown codes as "Catch = <n>") instead of the
  "forth-throw" sentinel. ABORT" text is carried as a structured
  payload and shown only when the -2 throw goes uncaught -- CATCH
  stays silent and the payload cannot go stale. ABORT throws -1
  through the same path.
- WORDS: optional same-line substring filter (WORDS FDEPTH), skips
  internal words (new INTERNAL header flag, set at create for
  underscore-prefixed names), wraps at 78 columns, prints a count.
  ORDER names wids (FORTH / wid#N) instead of Rust debug output.
- .S honors BASE. New: F.S (float stack), DUMP (hex+ASCII,
  bounds-checked, 4K cap), ? (fetch-and-print, boot.fth). BYE is a
  real word now: sets a VM flag the driver honors (exits REPL,
  stops rest of line/file).

CLI:
- Persistent history (~/.local/state/wafer/history, 0600 perms,
  $WAFER_HISTORY override), Tab completion over the live dictionary
  (snapshot refreshed after each line), Up/Down do prefix history
  search, Ctrl-C clears the line instead of exiting.

Web:
- History survives reloads (localStorage, cap 200, dedup, init-code
  runs excluded), User Words palette populated via new words()
  export, stack bar annotates non-decimal BASE, base() reads the
  real BASE sysvar instead of returning a hardcoded 10.
2026-08-05 14:57:12 +02:00
Oleksandr Kozachuk 4980648982 feat(bench): SwiftForth sf64 lane in cross-engine performance report
CI / check (push) Has been cancelled
sf64 discovery + stdin runner (no -e flag; input lines truncate at
~256 chars, so one statement per line), ucounter-based µs timing —
same wrapper shape as gforth utime. New sf64 + WAFER/sf columns,
informational only (no regression limit). Justfile: bench-compare
target; CARGO_PROFILE_RELEASE_STRIP=none for Darwin 27 dlopen bug.
2026-08-04 17:07:47 +02:00
Oleksandr Kozachuk 31dc6c6397 fix(core): no SystemTime on wasm32 — fixed boot seed + UTIME Forth error
CI / check (push) Has been cancelled
2026-07-29 16:56:31 +02:00
Oleksandr Kozachuk 35b78193fd feat(boot): add -ROT <= >= gforth extensions
CI / check (push) Has been cancelled
Non-standard but ubiquitous words; absence aborted otherwise-valid
gforth programs with unknown-word errors.
2026-07-18 15:40:39 +02:00
ok2 d5acdc0e7b fix: Rust 1.95 clippy — match guards + map_or
CI / check (push) Has been cancelled
Rust 1.95 promoted collapsible_match and map_unwrap_or; CI runs
-D warnings so they break the build. Collapse nested `if`s into
match guards across codegen/optimizer/export, and swap
map().unwrap_or(..) for map_or / is_ok_and.
2026-04-21 17:00:21 +02:00
25 changed files with 4279 additions and 445 deletions
+1 -1
View File
@@ -79,7 +79,7 @@ Handle in `interpret_token_immediate()` or `compile_token()` as a special case.
## Testing
- Run `cargo test --workspace` before committing (currently 431 unit + 1 benchmark + 11 compliance + 9 comparison)
- Run `cargo test --workspace` before committing (currently 542 unit + 1 benchmark + 11 compliance + 9 comparison + 5 crypto)
- Forth 2012 compliance: `cargo test -p wafer-core --test compliance`
- Cross-engine comparison (vs gforth): `cargo test -p wafer-core --test comparison`
- Performance benchmarks (release mode): `cargo test -p wafer-core --test comparison -- --nocapture --ignored`
+15
View File
@@ -43,6 +43,14 @@ bench:
bench-opts:
cargo test -p wafer-core --test benchmark_report -- --nocapture --ignored
# Cross-engine performance report: WAFER vs gforth vs SwiftForth (sf64)
bench-compare:
CARGO_PROFILE_RELEASE_STRIP=none cargo test -p wafer-core --release --test comparison -- --nocapture --ignored performance_report
# Cross-engine correctness lanes: program corpus vs gforth + sf64 oracles
compare-correctness:
cargo test -p wafer-core --test comparison -- --nocapture --ignored compare_all_programs
# Check dependency licenses and advisories
deny:
cargo deny check
@@ -58,6 +66,13 @@ ci: fmt clippy deny test
check:
cargo check --workspace
# Install the wafer CLI (release build) and bat syntax highlighting.
# STRIP=none: Cargo's release default (strip = "debuginfo") emits dylibs that
# macOS 27's dyld rejects ("mis-aligned LINKEDIT string pool"), so proc macros
# fail to load during the build itself.
install: install-syntax
CARGO_PROFILE_RELEASE_STRIP=none cargo install --path crates/cli --locked
# Install bat syntax highlighting for WAFER / Forth
install-syntax:
mkdir -p ~/.config/bat/syntaxes
+2 -1
View File
@@ -95,7 +95,7 @@ Times in microseconds. WAFER/gf < 1.0 means WAFER is faster. CONSOL = after `CON
## Testing
```bash
# All tests (~450 currently passing)
# All tests (~570 currently passing)
cargo test --workspace
# Forth 2012 compliance suite
@@ -185,6 +185,7 @@ Over 200 words are implemented across the following categories:
| Strings | `COMPARE SEARCH SLITERAL REPLACES SUBSTITUTE UNESCAPE` |
| Floating-Pt | `F+ F- F* F/ FABS FNEGATE FSQRT FSIN FCOS FTAN FEXP FLOG FMIN FMAX` and 55+ more |
| Case | `CASE OF ENDOF ENDCASE` |
| Tools | `WORDS SEE SEE-IR HELP INCLUDE INCLUDED .S F.S ? DUMP MARKER REMEMBER EMPTY GILD BYE` |
## Web REPL
+190 -58
View File
@@ -137,7 +137,9 @@ fn cmd_build(
) -> anyhow::Result<()> {
let source = std::fs::read_to_string(file)?;
let mut vm = ForthVM::<NativeRuntime>::new()?;
// Exported modules are production artifacts: no stack guards by default
let mut vm = ForthVM::<NativeRuntime>::new_with_config(vm_config(false))?;
vm.set_source_loader(fs_loader());
vm.set_recording(true);
vm.evaluate(&source)?;
@@ -260,18 +262,38 @@ fn cmd_run(file: &str) -> anyhow::Result<()> {
Ok(())
}
/// `WaferConfig` for CLI-created VMs. `WAFER_STACK_GUARDS=0|1` overrides
/// the per-command default (REPL/file execution on, build off).
fn vm_config(default_guards: bool) -> wafer_core::config::WaferConfig {
let mut cfg = wafer_core::config::WaferConfig::all();
cfg.codegen.stack_guards = match std::env::var("WAFER_STACK_GUARDS").ok().as_deref() {
Some("0") => false,
Some(_) => true,
None => default_guards,
};
cfg
}
/// Filesystem source loader for INCLUDE/INCLUDED.
fn fs_loader() -> Box<dyn Fn(&str) -> anyhow::Result<String> + Send + Sync> {
Box::new(|path| Ok(std::fs::read_to_string(path)?))
}
/// `wafer` (REPL) or `wafer program.fth` (evaluate and exit)
fn cmd_eval_or_repl(file: Option<&str>) -> anyhow::Result<()> {
let mut vm = ForthVM::<NativeRuntime>::new()?;
let mut vm = ForthVM::<NativeRuntime>::new_with_config(vm_config(true))?;
vm.set_source_loader(fs_loader());
match file {
Some(file) => {
let source = std::fs::read_to_string(file)?;
vm.evaluate(&source)?;
// Through the include machinery: file:line error context and a
// base directory for nested INCLUDEs.
let result = vm.include(file);
let output = vm.take_output();
if !output.is_empty() {
print!("{output}");
}
result?;
}
None => {
if !stdin_is_tty() {
@@ -285,66 +307,17 @@ fn cmd_eval_or_repl(file: Option<&str>) -> anyhow::Result<()> {
if !output.is_empty() {
print!("{output}");
}
if vm.bye_requested() {
break;
}
}
Err(e) => {
eprintln!("Error: {e}");
eprintln!("Error: {e:#}");
}
}
}
} else {
// Interactive REPL
println!(
"WAFER v{} - WebAssembly Forth Engine in Rust",
env!("CARGO_PKG_VERSION")
);
println!("Type BYE to exit.");
let mut rl = rustyline::DefaultEditor::new()?;
loop {
let prompt = if vm.is_compiling() { " ] " } else { "> " };
match rl.readline(prompt) {
Ok(line) => {
let trimmed = line.trim();
if trimmed.eq_ignore_ascii_case("BYE") {
break;
}
let _ = rl.add_history_entry(&line);
match vm.evaluate(&line) {
Ok(()) => {
let output = vm.take_output();
// PAGE (form feed) clears the terminal
if output.contains('\x0C') {
print!("\x1b[2J\x1b[H");
}
let output = output.replace('\x0C', "");
if !vm.is_compiling() {
// Move cursor back up to end of input line so
// output appears inline, like traditional Forth:
// > 2 2 + . 4 ok
let col = prompt.len() + line.len() + 1;
print!("\x1b[A\x1b[{col}G {output} ok");
println!();
} else if !output.is_empty() {
print!("{output}");
}
}
Err(e) => {
eprintln!("Error: {e}");
}
}
}
Err(
rustyline::error::ReadlineError::Interrupted
| rustyline::error::ReadlineError::Eof,
) => {
break;
}
Err(e) => {
eprintln!("Readline error: {e}");
break;
}
}
}
run_repl(&mut vm)?;
}
}
}
@@ -357,3 +330,162 @@ fn stdin_is_tty() -> bool {
use std::io::IsTerminal;
std::io::stdin().is_terminal()
}
/// Completes the token under the cursor against the live dictionary.
struct WaferHelper {
words: Vec<String>,
}
impl rustyline::completion::Completer for WaferHelper {
type Candidate = String;
fn complete(
&self,
line: &str,
pos: usize,
_ctx: &rustyline::Context<'_>,
) -> rustyline::Result<(usize, Vec<String>)> {
let start = line[..pos]
.rfind(|c: char| c.is_whitespace())
.map_or(0, |i| i + 1);
let prefix = line[start..pos].to_ascii_uppercase();
let mut matches: Vec<String> = self
.words
.iter()
.filter(|w| w.to_ascii_uppercase().starts_with(&prefix))
.cloned()
.collect();
matches.sort();
matches.dedup();
Ok((start, matches))
}
}
impl rustyline::hint::Hinter for WaferHelper {
type Hint = String;
}
impl rustyline::highlight::Highlighter for WaferHelper {}
impl rustyline::validate::Validator for WaferHelper {}
impl rustyline::Helper for WaferHelper {}
/// History file: `$WAFER_HISTORY`, else `$XDG_STATE_HOME/wafer/history`,
/// else `~/.local/state/wafer/history`.
fn history_path() -> Option<std::path::PathBuf> {
if let Some(p) = std::env::var_os("WAFER_HISTORY") {
return Some(p.into());
}
let base = std::env::var_os("XDG_STATE_HOME")
.map(std::path::PathBuf::from)
.or_else(|| {
std::env::var_os("HOME").map(|h| std::path::PathBuf::from(h).join(".local/state"))
})?;
Some(base.join("wafer/history"))
}
/// Interactive REPL: line editing, persistent history with prefix search
/// on Up/Down, and Tab completion over the live dictionary.
fn run_repl(vm: &mut ForthVM<NativeRuntime>) -> anyhow::Result<()> {
use rustyline::{Cmd, Editor, EventHandler, KeyCode, KeyEvent, Modifiers};
println!(
"WAFER v{} - WebAssembly Forth Engine in Rust",
env!("CARGO_PKG_VERSION")
);
println!("Type BYE to exit.");
let config = rustyline::Config::builder()
.completion_type(rustyline::CompletionType::List)
.history_ignore_dups(true)?
.build();
let mut rl: Editor<WaferHelper, rustyline::history::DefaultHistory> =
Editor::with_config(config)?;
rl.set_helper(Some(WaferHelper {
words: vm.word_names(),
}));
// Up/Down recall only entries starting with the typed prefix
rl.bind_sequence(
KeyEvent(KeyCode::Up, Modifiers::NONE),
EventHandler::Simple(Cmd::HistorySearchBackward),
);
rl.bind_sequence(
KeyEvent(KeyCode::Down, Modifiers::NONE),
EventHandler::Simple(Cmd::HistorySearchForward),
);
let history = history_path();
if let Some(path) = &history {
if let Some(dir) = path.parent() {
let _ = std::fs::create_dir_all(dir);
}
let _ = rl.load_history(path);
}
let save_history = |rl: &mut Editor<WaferHelper, rustyline::history::DefaultHistory>| {
if let Some(path) = &history {
let _ = rl.save_history(path);
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let _ = std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o600));
}
}
};
loop {
let prompt = if vm.is_compiling() { " ] " } else { "> " };
match rl.readline(prompt) {
Ok(line) => {
let _ = rl.add_history_entry(&line);
match vm.evaluate(&line) {
Ok(()) => {
let output = vm.take_output();
if vm.bye_requested() {
break;
}
// PAGE (form feed) clears the terminal
if output.contains('\x0C') {
print!("\x1b[2J\x1b[H");
}
let output = output.replace('\x0C', "");
if !vm.is_compiling() {
if output.contains('\n') {
// Multi-line output (DUMP, WORDS, ...):
// print as a block, then ok on its own line
print!("{output}");
if !output.ends_with('\n') {
println!();
}
println!(" ok");
} else {
// Move cursor back up to end of input line so
// output appears inline, like traditional Forth:
// > 2 2 + . 4 ok
let col = prompt.len() + line.len() + 1;
print!("\x1b[A\x1b[{col}G {output} ok");
println!();
}
} else if !output.is_empty() {
print!("{output}");
}
}
Err(e) => {
eprintln!("Error: {e:#}");
}
}
// New definitions may have appeared: refresh completion
if let Some(h) = rl.helper_mut() {
h.words = vm.word_names();
}
save_history(&mut rl);
}
// Ctrl-C abandons the current line, Ctrl-D exits
Err(rustyline::error::ReadlineError::Interrupted) => {}
Err(rustyline::error::ReadlineError::Eof) => break,
Err(e) => {
eprintln!("Readline error: {e}");
break;
}
}
}
save_history(&mut rl);
Ok(())
}
+33 -2
View File
@@ -72,6 +72,19 @@
1-
REPEAT ;
\ ---------------------------------------------------------------
\ Common extensions (not in Forth 2012, gforth-compatible)
\ ---------------------------------------------------------------
\ -ROT ( x1 x2 x3 -- x3 x1 x2 ) rotate top item to third place
: -ROT ROT ROT ;
\ <= ( n1 n2 -- flag ) true if n1 <= n2 (signed)
: <= > 0= ;
\ >= ( n1 n2 -- flag ) true if n1 >= n2 (signed)
: >= < 0= ;
\ ---------------------------------------------------------------
\ Phase 2: Double-cell arithmetic
\ ---------------------------------------------------------------
@@ -184,8 +197,8 @@
\ TYPE ( c-addr u -- ) output u characters
: TYPE 0 ?DO DUP C@ EMIT 1+ LOOP DROP ;
\ SPACES ( n -- ) output n spaces
: SPACES 0 ?DO SPACE LOOP ;
\ SPACES ( n -- ) output n spaces (nothing for n <= 0, per 6.1.2230)
: SPACES 0 MAX 0 ?DO SPACE LOOP ;
\ Pictured numeric output constants
\ PICT_BUF_TOP = 0x05C0 = 1472, SYSVAR_HLD = 28
@@ -230,6 +243,9 @@
\ U. ( u -- ) print unsigned number and space
: U. 0 <# #S #> TYPE SPACE ;
\ ? ( a-addr -- ) fetch and print
: ? @ . ;
\ .R ( n width -- ) print right-justified signed number
: .R >R DUP ABS 0 <# #S ROT SIGN #> R> OVER - SPACES TYPE ;
@@ -242,6 +258,21 @@
\ D.R ( d width -- ) print right-justified signed double
: D.R >R SWAP OVER DABS <# #S ROT SIGN #> R> OVER - SPACES TYPE ;
\ ---------------------------------------------------------------
\ Return-stack introspection (debug aids)
\ ---------------------------------------------------------------
\ RDEPTH ( -- n ) number of cells on the return stack
\ RETURN_STACK_TOP = 9728 (0x2600). Only >R temps and loop params
\ live there; return addresses are on the WASM call stack.
: RDEPTH 9728 RP@ - 2 RSHIFT ;
\ .RS ( -- ) print the return stack bottom-to-top, like .S
\ Walks with BEGIN/WHILE (not DO) so the walk itself never pushes
\ onto the return stack it is printing.
: .RS ." R:<" RDEPTH 0 .R ." > "
9728 BEGIN DUP RP@ > WHILE 4 - DUP @ . REPEAT DROP ;
\ ---------------------------------------------------------------
\ Phase 6: DEFER support
\ ---------------------------------------------------------------
+179 -31
View File
@@ -19,7 +19,10 @@ use wasm_encoder::{
use crate::dictionary::WordId;
use crate::error::{WaferError, WaferResult};
use crate::ir::IrOp;
use crate::memory::{CELL_SIZE, SYSVAR_LEAVE_FLAG};
use crate::memory::{
CELL_SIZE, DATA_STACK_BASE, DATA_STACK_TOP, FLOAT_STACK_BASE, FLOAT_STACK_TOP,
RETURN_STACK_BASE, RETURN_STACK_TOP, SYSVAR_FAULT_CODE, SYSVAR_LEAVE_FLAG,
};
// ---------------------------------------------------------------------------
// Import indices (order matters: imports numbered sequentially by kind)
@@ -94,6 +97,9 @@ pub struct CodegenConfig {
pub table_size: u32,
/// Enable stack-to-local promotion for straight-line words.
pub stack_to_local_promotion: bool,
/// Table index of the `_STACK_FAULT_` host word; `Some` enables
/// stack under/overflow guards in the emitted code.
pub stack_guards: Option<u32>,
}
/// Result of compiling a word to WASM.
@@ -109,16 +115,73 @@ pub struct CompiledModule {
// Instruction-level helpers (free functions that take &mut Function)
// ---------------------------------------------------------------------------
/// Decrement the cached `$dsp` local by `CELL_SIZE`.
// Stack-guard emission. The fault word's table index is stashed in a
// thread-local by `compile_word` (None = guards off) so the low-level
// push/pop helpers can stay plain `&mut Function` free functions
// without threading config through every emitter.
thread_local! {
static GUARD_FAULT: std::cell::Cell<Option<u32>> = const { std::cell::Cell::new(None) };
}
/// With guards on: emit `if <cond> { mem[SYSVAR_FAULT_CODE] = code;
/// call _STACK_FAULT_ }`. `cond` must leave an i32 boolean on the
/// operand stack. The fault host word throws, so the `if` never falls
/// through on the failure path.
fn emit_guard(f: &mut Function, code: i32, cond: impl FnOnce(&mut Function)) {
let Some(fault_idx) = GUARD_FAULT.get() else {
return;
};
cond(f);
f.instruction(&Instruction::If(BlockType::Empty))
.instruction(&Instruction::I32Const(SYSVAR_FAULT_CODE as i32))
.instruction(&Instruction::I32Const(code))
.instruction(&Instruction::I32Store(MEM4))
.instruction(&Instruction::I32Const(fault_idx as i32))
.instruction(&Instruction::CallIndirect {
type_index: TYPE_VOID,
table_index: TABLE,
})
.instruction(&Instruction::End);
}
/// Guard: data stack has at least `n` cells (else throw -4).
fn guard_dsp_underflow(f: &mut Function, n: u32) {
emit_guard(f, -4, |f| {
f.instruction(&Instruction::LocalGet(CACHED_DSP_LOCAL))
.instruction(&Instruction::I32Const((n * CELL_SIZE) as i32))
.instruction(&Instruction::I32Add)
.instruction(&Instruction::I32Const(DATA_STACK_TOP as i32))
.instruction(&Instruction::I32GtU);
});
}
/// Guard: data stack has room for `n` more cells (else throw -3).
fn guard_dsp_overflow(f: &mut Function, n: u32) {
emit_guard(f, -3, |f| {
f.instruction(&Instruction::LocalGet(CACHED_DSP_LOCAL))
.instruction(&Instruction::I32Const(
(DATA_STACK_BASE + n * CELL_SIZE) as i32,
))
.instruction(&Instruction::I32LtU);
});
}
/// Decrement the cached `$dsp` local by `CELL_SIZE` (allocate one cell).
/// This is the single choke point for data-stack pushes, so the
/// overflow guard lives here.
fn dsp_dec(f: &mut Function) {
guard_dsp_overflow(f, 1);
f.instruction(&Instruction::LocalGet(CACHED_DSP_LOCAL))
.instruction(&Instruction::I32Const(CELL_SIZE as i32))
.instruction(&Instruction::I32Sub)
.instruction(&Instruction::LocalSet(CACHED_DSP_LOCAL));
}
/// Increment the cached `$dsp` local by `CELL_SIZE`.
/// Increment the cached `$dsp` local by `CELL_SIZE` (free one cell).
/// Single choke point for data-stack pops (`DROP` never loads the
/// value, so the underflow guard must sit here, not in `pop`).
fn dsp_inc(f: &mut Function) {
guard_dsp_underflow(f, 1);
f.instruction(&Instruction::LocalGet(CACHED_DSP_LOCAL))
.instruction(&Instruction::I32Const(CELL_SIZE as i32))
.instruction(&Instruction::I32Add)
@@ -160,6 +223,7 @@ fn pop_to(f: &mut Function, local: u32) {
/// Read the top of the data stack without popping (value on operand stack).
fn peek(f: &mut Function) {
guard_dsp_underflow(f, 1);
f.instruction(&Instruction::LocalGet(CACHED_DSP_LOCAL))
.instruction(&Instruction::I32Load(MEM4));
}
@@ -182,6 +246,13 @@ fn dsp_reload(f: &mut Function) {
/// Push a value from the WASM operand stack onto the return stack via `tmp`.
fn rpush_via_local(f: &mut Function, tmp: u32) {
emit_guard(f, -5, |f| {
f.instruction(&Instruction::GlobalGet(RSP))
.instruction(&Instruction::I32Const(
(RETURN_STACK_BASE + CELL_SIZE) as i32,
))
.instruction(&Instruction::I32LtU);
});
f.instruction(&Instruction::LocalSet(tmp));
// rsp -= CELL_SIZE
f.instruction(&Instruction::GlobalGet(RSP))
@@ -194,8 +265,18 @@ fn rpush_via_local(f: &mut Function, tmp: u32) {
.instruction(&Instruction::I32Store(MEM4));
}
/// Guard: return stack is non-empty (else throw -6).
fn guard_rsp_underflow(f: &mut Function) {
emit_guard(f, -6, |f| {
f.instruction(&Instruction::GlobalGet(RSP))
.instruction(&Instruction::I32Const(RETURN_STACK_TOP as i32))
.instruction(&Instruction::I32GeU);
});
}
/// Pop the return stack onto the WASM operand stack.
fn rpop(f: &mut Function) {
guard_rsp_underflow(f);
f.instruction(&Instruction::GlobalGet(RSP))
.instruction(&Instruction::I32Load(MEM4));
// rsp += CELL_SIZE
@@ -207,6 +288,7 @@ fn rpop(f: &mut Function) {
/// Peek at the top of the return stack (no pop).
fn rpeek(f: &mut Function) {
guard_rsp_underflow(f);
f.instruction(&Instruction::GlobalGet(RSP))
.instruction(&Instruction::I32Load(MEM4));
}
@@ -253,7 +335,9 @@ struct EmitCtx {
}
/// Decrement the FSP global by 8 (allocate space for one f64).
/// Single choke point for float pushes: overflow guard lives here.
fn fsp_dec(f: &mut Function) {
guard_fsp_overflow(f);
f.instruction(&Instruction::GlobalGet(FSP))
.instruction(&Instruction::I32Const(8))
.instruction(&Instruction::I32Sub)
@@ -261,7 +345,10 @@ fn fsp_dec(f: &mut Function) {
}
/// Increment the FSP global by 8 (free space for one f64).
/// Single choke point for float pops (`FDROP` never loads the value):
/// underflow guard lives here.
fn fsp_inc(f: &mut Function) {
guard_fsp_underflow(f);
f.instruction(&Instruction::GlobalGet(FSP))
.instruction(&Instruction::I32Const(8))
.instruction(&Instruction::I32Add)
@@ -278,6 +365,24 @@ fn fpush_via_local(f: &mut Function, tmp: u32) {
.instruction(&Instruction::F64Store(MEM8));
}
/// Guard: float stack has room for one more f64 (else throw -44).
fn guard_fsp_overflow(f: &mut Function) {
emit_guard(f, -44, |f| {
f.instruction(&Instruction::GlobalGet(FSP))
.instruction(&Instruction::I32Const((FLOAT_STACK_BASE + 8) as i32))
.instruction(&Instruction::I32LtU);
});
}
/// Guard: float stack is non-empty (else throw -45).
fn guard_fsp_underflow(f: &mut Function) {
emit_guard(f, -45, |f| {
f.instruction(&Instruction::GlobalGet(FSP))
.instruction(&Instruction::I32Const(FLOAT_STACK_TOP as i32))
.instruction(&Instruction::I32GeU);
});
}
/// Decrement FSP, then store the f64 from local `src` at [FSP].
fn fpush_from_local(f: &mut Function, src: u32) {
fsp_dec(f);
@@ -295,6 +400,7 @@ fn fpop(f: &mut Function) {
/// Load f64 from [FSP] onto the WASM operand stack without popping.
fn fpeek(f: &mut Function) {
guard_fsp_underflow(f);
f.instruction(&Instruction::GlobalGet(FSP))
.instruction(&Instruction::F64Load(MEM8));
}
@@ -793,9 +899,20 @@ fn emit_op(f: &mut Function, op: &IrOp, ctx: &mut EmitCtx) {
.instruction(&Instruction::I32Store(MEM4));
}
IrOp::RpFetch => {
// Push the current return-stack pointer onto the data stack.
// `$rsp` lives in a global (not cached), so no writeback needed.
dsp_dec(f);
f.instruction(&Instruction::LocalGet(CACHED_DSP_LOCAL))
.instruction(&Instruction::GlobalGet(RSP))
.instruction(&Instruction::I32Store(MEM4));
}
// -- Compound operations -----------------------------------------------
IrOp::TwoDup => {
// ( a b -- a b a b )
guard_dsp_underflow(f, 2);
guard_dsp_overflow(f, 2);
f.instruction(&Instruction::LocalGet(CACHED_DSP_LOCAL))
.instruction(&Instruction::I32Load(MEM4)); // b
f.instruction(&Instruction::LocalSet(SCRATCH_BASE));
@@ -822,6 +939,7 @@ fn emit_op(f: &mut Function, op: &IrOp, ctx: &mut EmitCtx) {
IrOp::TwoDrop => {
// ( a b -- )
guard_dsp_underflow(f, 2);
f.instruction(&Instruction::LocalGet(CACHED_DSP_LOCAL))
.instruction(&Instruction::I32Const((CELL_SIZE * 2) as i32))
.instruction(&Instruction::I32Add)
@@ -1133,7 +1251,9 @@ fn is_promotable(ops: &[IrOp]) -> bool {
fn is_promotable_body(ops: &[IrOp]) -> bool {
for op in ops {
match op {
IrOp::Call(_) | IrOp::TailCall(_) | IrOp::Execute | IrOp::SpFetch => return false,
IrOp::Call(_) | IrOp::TailCall(_) | IrOp::Execute | IrOp::SpFetch | IrOp::RpFetch => {
return false;
}
IrOp::ToR | IrOp::FromR | IrOp::Exit => return false,
IrOp::ForthLocalGet(_) | IrOp::ForthLocalSet(_) => return false,
IrOp::ForthFLocalGet(_) | IrOp::ForthFLocalSet(_) => return false,
@@ -1545,6 +1665,11 @@ impl StackSim {
/// Emit the promoted prologue: load `preload` items from the memory stack
/// into WASM locals.
fn emit_promoted_prologue(f: &mut Function, preload: u32, sim: &mut StackSim) {
// One entry check covers the whole promoted word: the caller must
// have at least `preload` cells on the data stack.
if preload > 0 {
guard_dsp_underflow(f, preload);
}
// Load items: mem[dsp] = top of stack, mem[dsp+4] = second, etc.
// We load them top-first, then reverse the sim stack so that
// sim.stack[0] = deepest loaded, sim.stack[last] = top.
@@ -1575,6 +1700,7 @@ fn emit_promoted_prologue(f: &mut Function, preload: u32, sim: &mut StackSim) {
fn emit_promoted_epilogue(f: &mut Function, sim: &mut StackSim) {
let remaining = sim.stack.len() as u32;
if remaining > 0 {
guard_dsp_overflow(f, remaining);
// Decrement cached DSP for the items we're pushing back
f.instruction(&Instruction::LocalGet(CACHED_DSP_LOCAL));
f.instruction(&Instruction::I32Const((remaining * CELL_SIZE) as i32));
@@ -2012,14 +2138,12 @@ fn emit_promoted_op(f: &mut Function, op: &IrOp, sim: &mut StackSim) {
// Outside loops, RFetch shouldn't appear in promoted code
}
IrOp::LoopJ => {
if sim.loop_index_stack.len() >= 2 {
let (outer_index, _) = sim.loop_index_stack[sim.loop_index_stack.len() - 2];
let result = sim.alloc();
f.instruction(&Instruction::LocalGet(outer_index));
f.instruction(&Instruction::LocalSet(result));
sim.push(result);
}
IrOp::LoopJ if sim.loop_index_stack.len() >= 2 => {
let (outer_index, _) = sim.loop_index_stack[sim.loop_index_stack.len() - 2];
let result = sim.alloc();
f.instruction(&Instruction::LocalGet(outer_index));
f.instruction(&Instruction::LocalSet(result));
sim.push(result);
}
IrOp::Exit => {
@@ -2147,15 +2271,15 @@ fn needs_f64_locals(ops: &[IrOp]) -> bool {
return true;
}
}
IrOp::DoLoop { body, .. } | IrOp::BeginUntil { body } | IrOp::BeginAgain { body } => {
if needs_f64_locals(body) {
return true;
}
IrOp::DoLoop { body, .. } | IrOp::BeginUntil { body } | IrOp::BeginAgain { body }
if needs_f64_locals(body) =>
{
return true;
}
IrOp::BeginWhileRepeat { test, body } => {
if needs_f64_locals(test) || needs_f64_locals(body) {
return true;
}
IrOp::BeginWhileRepeat { test, body }
if needs_f64_locals(test) || needs_f64_locals(body) =>
{
return true;
}
IrOp::BeginDoubleWhileRepeat {
outer_test,
@@ -2193,6 +2317,9 @@ fn body_needs_return_stack(ops: &[IrOp]) -> bool {
match op {
IrOp::Call(_) | IrOp::TailCall(_) | IrOp::Execute => return true,
IrOp::ToR | IrOp::FromR => return true,
// RP@ observes the return stack, so loop params must be there
// (otherwise inlined RDEPTH/.RS would report an empty stack).
IrOp::RpFetch => return true,
// RFetch (I) is handled by loop locals in the fast path — not a problem.
// LoopJ is also handled by loop locals.
// Only explicit >R / R> / calls force the slow path.
@@ -2209,15 +2336,15 @@ fn body_needs_return_stack(ops: &[IrOp]) -> bool {
return true;
}
}
IrOp::DoLoop { body, .. } | IrOp::BeginUntil { body } | IrOp::BeginAgain { body } => {
if body_needs_return_stack(body) {
return true;
}
IrOp::DoLoop { body, .. } | IrOp::BeginUntil { body } | IrOp::BeginAgain { body }
if body_needs_return_stack(body) =>
{
return true;
}
IrOp::BeginWhileRepeat { test, body } => {
if body_needs_return_stack(test) || body_needs_return_stack(body) {
return true;
}
IrOp::BeginWhileRepeat { test, body }
if body_needs_return_stack(test) || body_needs_return_stack(body) =>
{
return true;
}
IrOp::BeginDoubleWhileRepeat {
outer_test,
@@ -2405,10 +2532,13 @@ fn count_forth_f_locals(ops: &[IrOp]) -> u32 {
/// This is the JIT path: each word gets its own module that imports
/// shared memory, globals, and function table from the host.
pub fn compile_word(
_name: &str,
name: &str,
body: &[IrOp],
config: &CodegenConfig,
) -> WaferResult<CompiledModule> {
// Arm (or disarm) stack-guard emission for this compilation.
GUARD_FAULT.set(config.stack_guards);
let mut module = Module::new();
// -- Type section --
@@ -2569,6 +2699,16 @@ pub fn compile_word(
code.function(&func);
module.section(&code);
// -- Name section: carries the Forth word name into wasmtime trap
// backtraces (best-effort symbolication, WS-008).
let mut names = wasm_encoder::NameSection::new();
names.module(name);
let mut fn_names = wasm_encoder::NameMap::new();
fn_names.append(0, "emit");
fn_names.append(WORD_FUNC, name);
names.functions(&fn_names);
module.section(&names);
let bytes = module.finish();
// Validate
@@ -2873,8 +3013,9 @@ pub fn compile_consolidated_module(
words: &[(WordId, Vec<IrOp>)],
local_fn_map: &HashMap<WordId, u32>,
table_size: u32,
stack_guards: Option<u32>,
) -> WaferResult<Vec<u8>> {
compile_multi_word_module(words, local_fn_map, table_size, None)
compile_multi_word_module(words, local_fn_map, table_size, None, stack_guards)
}
/// Compile an exportable WASM module with embedded memory and metadata.
@@ -2887,8 +3028,9 @@ pub fn compile_exportable_module(
local_fn_map: &HashMap<WordId, u32>,
table_size: u32,
export: &ExportSections<'_>,
stack_guards: Option<u32>,
) -> WaferResult<Vec<u8>> {
compile_multi_word_module(words, local_fn_map, table_size, Some(export))
compile_multi_word_module(words, local_fn_map, table_size, Some(export), stack_guards)
}
/// Internal: build a multi-word WASM module. When `export` is `Some`, adds
@@ -2898,7 +3040,11 @@ fn compile_multi_word_module(
local_fn_map: &HashMap<WordId, u32>,
table_size: u32,
export: Option<&ExportSections<'_>>,
stack_guards: Option<u32>,
) -> WaferResult<Vec<u8>> {
// Arm (or disarm) stack-guard emission for this module.
GUARD_FAULT.set(stack_guards);
let has_data = export.is_some_and(|e| !e.memory_snapshot.is_empty());
let mut module = Module::new();
@@ -3127,6 +3273,7 @@ mod tests {
base_fn_index: 0,
table_size: 16,
stack_to_local_promotion: true,
stack_guards: None,
}
}
@@ -3351,6 +3498,7 @@ mod tests {
base_fn_index: 7,
table_size: 16,
stack_to_local_promotion: true,
stack_guards: None,
};
let m = compile_word("t", &[IrOp::PushI32(1)], &cfg).unwrap();
assert_eq!(m.fn_index, 7);
+7
View File
@@ -7,6 +7,11 @@ use crate::optimizer::OptConfig;
pub struct CodegenOpts {
/// Enable stack-to-local promotion for straight-line words.
pub stack_to_local_promotion: bool,
/// Emit stack under/overflow guards in compiled words. Faults throw
/// standard codes (-3/-4/-5/-6/-44/-45) instead of silently
/// corrupting stack pointers. On by default; benchmarks and
/// exported production modules turn it off.
pub stack_guards: bool,
}
/// Master configuration for all WAFER optimizations.
@@ -32,6 +37,7 @@ impl WaferConfig {
},
codegen: CodegenOpts {
stack_to_local_promotion: true,
stack_guards: true,
},
}
}
@@ -49,6 +55,7 @@ impl WaferConfig {
},
codegen: CodegenOpts {
stack_to_local_promotion: false,
stack_guards: false,
},
}
}
+9 -9
View File
@@ -21,7 +21,7 @@ mod tests {
// Empty word list should produce nothing (but we guard against this at call site)
let words = vec![];
let map = HashMap::new();
let result = compile_consolidated_module(&words, &map, 16);
let result = compile_consolidated_module(&words, &map, 16, None);
// Empty is valid -- should produce a valid module with no functions
assert!(result.is_ok());
}
@@ -31,7 +31,7 @@ mod tests {
let words = vec![(WordId(1), vec![IrOp::PushI32(42)])];
let mut map = HashMap::new();
map.insert(WordId(1), 1u32); // function index 1 (after emit import)
let result = compile_consolidated_module(&words, &map, 16);
let result = compile_consolidated_module(&words, &map, 16, None);
assert!(result.is_ok());
}
@@ -49,7 +49,7 @@ mod tests {
map.insert(WordId(1), 1u32);
map.insert(WordId(2), 2u32);
map.insert(WordId(3), 3u32);
let result = compile_consolidated_module(&words, &map, 16);
let result = compile_consolidated_module(&words, &map, 16, None);
assert!(result.is_ok());
}
@@ -59,7 +59,7 @@ mod tests {
let words = vec![(WordId(3), vec![IrOp::Call(WordId(99))])];
let mut map = HashMap::new();
map.insert(WordId(3), 1u32);
let result = compile_consolidated_module(&words, &map, 256);
let result = compile_consolidated_module(&words, &map, 256, None);
assert!(result.is_ok());
}
@@ -72,7 +72,7 @@ mod tests {
let mut map = HashMap::new();
map.insert(WordId(1), 1u32);
map.insert(WordId(2), 2u32);
let result = compile_consolidated_module(&words, &map, 16);
let result = compile_consolidated_module(&words, &map, 16, None);
assert!(result.is_ok());
}
@@ -95,7 +95,7 @@ mod tests {
let mut map = HashMap::new();
map.insert(WordId(1), 1u32);
map.insert(WordId(2), 2u32);
let result = compile_consolidated_module(&words, &map, 16);
let result = compile_consolidated_module(&words, &map, 16, None);
assert!(result.is_ok());
}
@@ -120,7 +120,7 @@ mod tests {
let mut map = HashMap::new();
map.insert(WordId(1), 1u32);
map.insert(WordId(2), 2u32);
let result = compile_consolidated_module(&words, &map, 16);
let result = compile_consolidated_module(&words, &map, 16, None);
assert!(result.is_ok());
}
@@ -141,7 +141,7 @@ mod tests {
let mut map = HashMap::new();
map.insert(WordId(1), 1u32);
map.insert(WordId(2), 2u32);
let result = compile_consolidated_module(&words, &map, 16);
let result = compile_consolidated_module(&words, &map, 16, None);
assert!(result.is_ok());
}
@@ -163,7 +163,7 @@ mod tests {
let mut map = HashMap::new();
map.insert(WordId(1), 1u32);
map.insert(WordId(2), 2u32);
let result = compile_consolidated_module(&words, &map, 16);
let result = compile_consolidated_module(&words, &map, 16, None);
assert!(result.is_ok());
}
}
+31 -5
View File
@@ -18,6 +18,8 @@ pub mod flags {
pub const IMMEDIATE: u8 = 0x80;
/// Word is hidden (being compiled, not yet findable).
pub const HIDDEN: u8 = 0x40;
/// Word is an implementation detail: findable, but skipped by WORDS.
pub const INTERNAL: u8 = 0x20;
/// Mask for the name length (lower 5 bits).
pub const LENGTH_MASK: u8 = 0x1F;
/// Maximum word name length.
@@ -95,11 +97,17 @@ impl Dictionary {
// Write link field (points to previous LATEST)
self.write_u32_unchecked(entry_start, self.latest);
// Write flags byte: HIDDEN | length, optionally IMMEDIATE
// Write flags byte: HIDDEN | length, optionally IMMEDIATE.
// Underscore-prefixed names are implementation details by repo
// convention (see tools/editor-support): flag them INTERNAL so
// WORDS and completion skip them while FIND still works.
let mut flag_byte = flags::HIDDEN | (name_len as u8 & flags::LENGTH_MASK);
if immediate {
flag_byte |= flags::IMMEDIATE;
}
if name_bytes.first() == Some(&b'_') {
flag_byte |= flags::INTERNAL;
}
self.memory[(entry_start + 4) as usize] = flag_byte;
// Write name bytes
@@ -410,8 +418,21 @@ impl Dictionary {
}
/// Return names of all visible (non-hidden) words, newest first.
pub fn visible_words(&self) -> Vec<String> {
let mut names = Vec::new();
/// With `include_internal` false, words flagged INTERNAL are skipped.
pub fn visible_words(&self, include_internal: bool) -> Vec<String> {
self.visible_entries()
.into_iter()
.filter(|(_, _, internal)| include_internal || !internal)
.map(|(name, _, _)| name)
.collect()
}
/// All visible (non-hidden) entries, newest first:
/// (name, wordlist id, INTERNAL flag). The wid comes from the hash
/// index (entries themselves store no wid); words missing from the
/// index default to wid 1 (FORTH).
pub fn visible_entries(&self) -> Vec<(String, u32, bool)> {
let mut entries = Vec::new();
let mut addr = self.latest;
while addr != 0 {
let flags_byte = self.memory[(addr + 4) as usize];
@@ -420,7 +441,12 @@ impl Dictionary {
let name_start = (addr + 5) as usize;
let name = String::from_utf8_lossy(&self.memory[name_start..name_start + name_len])
.to_string();
names.push(name);
let wid = self
.index
.get(&name)
.and_then(|es| es.iter().find(|e| e.1 == addr))
.map_or(1, |e| e.0);
entries.push((name, wid, flags_byte & flags::INTERNAL != 0));
}
let link = self.read_u32_unchecked(addr);
if link == addr {
@@ -428,7 +454,7 @@ impl Dictionary {
}
addr = link;
}
names
entries
}
/// Get a reference to the raw memory buffer.
+7
View File
@@ -61,6 +61,13 @@ pub enum WaferError {
#[error("{0}")]
Abort(String),
/// An uncaught Forth THROW as reported to the user. `message` is the
/// full display text (standard message or ABORT" payload); `code`
/// carries the THROW code for typed consumers (CLI exit paths, web
/// REPL styling) via `Error::downcast_ref`.
#[error("{message}")]
UncaughtThrow { code: i32, message: String },
}
/// Result type alias for WAFER operations.
+10 -6
View File
@@ -120,8 +120,14 @@ pub fn export_module(
metadata_json: metadata_json.as_bytes(),
};
let wasm_bytes = compile_exportable_module(&words, &local_fn_map, table_size, &export_sections)
.map_err(|e| anyhow::anyhow!("export codegen error: {e}"))?;
let wasm_bytes = compile_exportable_module(
&words,
&local_fn_map,
table_size,
&export_sections,
vm.stack_guard_param(),
)
.map_err(|e| anyhow::anyhow!("export codegen error: {e}"))?;
Ok((wasm_bytes, metadata))
}
@@ -131,10 +137,8 @@ pub fn export_module(
fn collect_external_calls(ops: &[IrOp], ir_ids: &HashSet<WordId>, host_ids: &mut HashSet<WordId>) {
for op in ops {
match op {
IrOp::Call(id) | IrOp::TailCall(id) => {
if !ir_ids.contains(id) {
host_ids.insert(*id);
}
IrOp::Call(id) | IrOp::TailCall(id) if !ir_ids.contains(id) => {
host_ids.insert(*id);
}
IrOp::If {
then_body,
+2
View File
@@ -159,6 +159,8 @@ pub enum IrOp {
Execute,
/// Push the current data-stack pointer: ( -- addr )
SpFetch,
/// Push the current return-stack pointer: ( -- addr )
RpFetch,
// -- Float stack manipulation --
/// Float duplicate: ( F: r -- r r )
+2
View File
@@ -24,6 +24,8 @@ pub mod ir;
pub mod memory;
pub mod optimizer;
pub mod runtime;
pub mod see;
pub mod wordhelp;
// Outer interpreter: runtime-agnostic, works with any Runtime impl
#[allow(trivial_numeric_casts, clippy::unnecessary_cast)]
+2
View File
@@ -106,6 +106,8 @@ pub const SYSVAR_NUM_TIB: u32 = SYSVAR_BASE + 24;
pub const SYSVAR_HLD: u32 = SYSVAR_BASE + 28;
/// LEAVE flag: nonzero when LEAVE has been called inside a DO loop.
pub const SYSVAR_LEAVE_FLAG: u32 = SYSVAR_BASE + 32;
/// Throw code left by a compiled stack-guard fault for `_STACK_FAULT_`.
pub const SYSVAR_FAULT_CODE: u32 = SYSVAR_BASE + 36;
#[cfg(test)]
mod tests {
+14 -16
View File
@@ -591,15 +591,15 @@ fn contains_call_to(ops: &[IrOp], target: WordId) -> bool {
return true;
}
}
IrOp::DoLoop { body, .. } | IrOp::BeginUntil { body } | IrOp::BeginAgain { body } => {
if contains_call_to(body, target) {
return true;
}
IrOp::DoLoop { body, .. } | IrOp::BeginUntil { body } | IrOp::BeginAgain { body }
if contains_call_to(body, target) =>
{
return true;
}
IrOp::BeginWhileRepeat { test, body } => {
if contains_call_to(test, target) || contains_call_to(body, target) {
return true;
}
IrOp::BeginWhileRepeat { test, body }
if contains_call_to(test, target) || contains_call_to(body, target) =>
{
return true;
}
IrOp::BeginDoubleWhileRepeat {
outer_test,
@@ -651,15 +651,13 @@ fn contains_exit(ops: &[IrOp]) -> bool {
return true;
}
}
IrOp::DoLoop { body, .. } | IrOp::BeginUntil { body } | IrOp::BeginAgain { body } => {
if contains_exit(body) {
return true;
}
IrOp::DoLoop { body, .. } | IrOp::BeginUntil { body } | IrOp::BeginAgain { body }
if contains_exit(body) =>
{
return true;
}
IrOp::BeginWhileRepeat { test, body } => {
if contains_exit(test) || contains_exit(body) {
return true;
}
IrOp::BeginWhileRepeat { test, body } if contains_exit(test) || contains_exit(body) => {
return true;
}
_ => {}
}
+1736 -219
View File
File diff suppressed because it is too large Load Diff
+21 -2
View File
@@ -98,11 +98,29 @@ impl HostAccess for CallerHostAccess<'_, '_> {
let func = *func_ref
.unwrap_func()
.ok_or_else(|| anyhow::anyhow!("call_func: null funcref {fn_index}"))?;
func.call(&mut *self.caller, &[], &mut [])?;
func.call(&mut *self.caller, &[], &mut [])
.map_err(name_trap_frame)?;
Ok(())
}
}
/// Prefix a wasmtime trap error with the innermost named WASM frame.
/// Compiled words carry their Forth name in the module name section, so a
/// genuine trap reads "in <WORD>: wasm trap: ...". THROW-driven unwinds
/// also pass through here, but CATCH and `describe_uncaught` key on the
/// shared `throw_code` cell, never on the message, so the wrap is inert
/// for them.
fn name_trap_frame(e: wasmtime::Error) -> wasmtime::Error {
let name = e
.downcast_ref::<wasmtime::WasmBacktrace>()
.and_then(|bt| bt.frames().iter().find_map(|f| f.func_name()))
.map(str::to_string);
match name {
Some(n) => e.context(format!("in {n}")),
None => e,
}
}
/// Wasmtime-based native runtime.
pub struct NativeRuntime {
engine: Engine,
@@ -293,7 +311,8 @@ impl Runtime for NativeRuntime {
let func = *r
.unwrap_func()
.ok_or_else(|| anyhow::anyhow!("word {fn_index} is null funcref"))?;
func.call(&mut self.store, &[], &mut [])?;
func.call(&mut self.store, &[], &mut [])
.map_err(name_trap_frame)?;
Ok(())
}
+376
View File
@@ -0,0 +1,376 @@
//! IR pretty-printer for `SEE-IR` and the `SEE` fallback path.
//!
//! Renders a post-optimization IR body as indented, one-op-per-line text.
//! Simple ops print as short lowercase mnemonics (Forth glyphs where they
//! are universally recognizable: `@`, `!`, `0=`, `>r`, ...); structured ops
//! print as Forth control words with 2-space indented bodies. Calls resolve
//! `WordId`s to names through an optional resolver so the formatter itself
//! stays independent of the VM.
use crate::dictionary::WordId;
use crate::ir::IrOp;
/// Format an IR body as indented, one-op-per-line text.
pub fn format_ir(ops: &[IrOp]) -> String {
format_ir_with(ops, &|_| None)
}
/// Like [`format_ir`], resolving `Call`/`TailCall`/`Execute` targets to word
/// names via `resolve`; unresolved ids print as `#N`.
pub fn format_ir_with(ops: &[IrOp], resolve: &dyn Fn(WordId) -> Option<String>) -> String {
let mut out = String::new();
write_ops(&mut out, ops, 0, resolve);
out
}
fn line(out: &mut String, depth: usize, text: &str) {
for _ in 0..depth {
out.push_str(" ");
}
out.push_str(text);
out.push('\n');
}
fn callee(id: WordId, resolve: &dyn Fn(WordId) -> Option<String>) -> String {
resolve(id).unwrap_or_else(|| format!("#{}", id.0))
}
fn write_ops(
out: &mut String,
ops: &[IrOp],
depth: usize,
resolve: &dyn Fn(WordId) -> Option<String>,
) {
for op in ops {
write_op(out, op, depth, resolve);
}
}
fn write_op(out: &mut String, op: &IrOp, depth: usize, resolve: &dyn Fn(WordId) -> Option<String>) {
// Exhaustive on purpose: a new IrOp variant must show up here at
// compile time, not silently render wrong.
let simple: String = match op {
// -- Literals --
IrOp::PushI32(v) => format!("push {v}"),
IrOp::PushI64(v) => format!("push64 {v}"),
IrOp::PushF64(v) => format!("fpush {v}"),
// -- Stack manipulation --
IrOp::Drop => "drop".into(),
IrOp::Dup => "dup".into(),
IrOp::Swap => "swap".into(),
IrOp::Over => "over".into(),
IrOp::Rot => "rot".into(),
IrOp::Nip => "nip".into(),
IrOp::Tuck => "tuck".into(),
IrOp::TwoDup => "2dup".into(),
IrOp::TwoDrop => "2drop".into(),
// -- Arithmetic --
IrOp::Add => "add".into(),
IrOp::Sub => "sub".into(),
IrOp::Mul => "mul".into(),
IrOp::DivMod => "divmod".into(),
IrOp::Negate => "negate".into(),
IrOp::Abs => "abs".into(),
// -- Comparison --
IrOp::Eq => "eq".into(),
IrOp::NotEq => "ne".into(),
IrOp::Lt => "lt".into(),
IrOp::Gt => "gt".into(),
IrOp::LtUnsigned => "u<".into(),
IrOp::ZeroEq => "0=".into(),
IrOp::ZeroLt => "0<".into(),
// -- Logic --
IrOp::And => "and".into(),
IrOp::Or => "or".into(),
IrOp::Xor => "xor".into(),
IrOp::Invert => "invert".into(),
IrOp::Lshift => "lshift".into(),
IrOp::Rshift => "rshift".into(),
IrOp::ArithRshift => "arshift".into(),
// -- Memory --
IrOp::Fetch => "@".into(),
IrOp::Store => "!".into(),
IrOp::CFetch => "c@".into(),
IrOp::CStore => "c!".into(),
IrOp::PlusStore => "+!".into(),
// -- Calls --
IrOp::Call(id) => format!("call {}", callee(*id, resolve)),
IrOp::TailCall(id) => format!("tail-call {}", callee(*id, resolve)),
// -- Structured control flow (multi-line) --
IrOp::If {
then_body,
else_body,
} => {
line(out, depth, "if");
write_ops(out, then_body, depth + 1, resolve);
if let Some(eb) = else_body {
line(out, depth, "else");
write_ops(out, eb, depth + 1, resolve);
}
line(out, depth, "then");
return;
}
IrOp::DoLoop { body, is_plus_loop } => {
line(out, depth, "do");
write_ops(out, body, depth + 1, resolve);
line(out, depth, if *is_plus_loop { "+loop" } else { "loop" });
return;
}
IrOp::BeginUntil { body } => {
line(out, depth, "begin");
write_ops(out, body, depth + 1, resolve);
line(out, depth, "until");
return;
}
IrOp::BeginAgain { body } => {
line(out, depth, "begin");
write_ops(out, body, depth + 1, resolve);
line(out, depth, "again");
return;
}
IrOp::BeginWhileRepeat { test, body } => {
line(out, depth, "begin");
write_ops(out, test, depth + 1, resolve);
line(out, depth, "while");
write_ops(out, body, depth + 1, resolve);
line(out, depth, "repeat");
return;
}
IrOp::BeginDoubleWhileRepeat {
outer_test,
inner_test,
body,
after_repeat,
else_body,
} => {
line(out, depth, "begin");
write_ops(out, outer_test, depth + 1, resolve);
line(out, depth, "while");
write_ops(out, inner_test, depth + 1, resolve);
line(out, depth, "while");
write_ops(out, body, depth + 1, resolve);
line(out, depth, "repeat");
write_ops(out, after_repeat, depth + 1, resolve);
if let Some(eb) = else_body {
line(out, depth, "else");
write_ops(out, eb, depth + 1, resolve);
}
line(out, depth, "then");
return;
}
IrOp::Exit => "exit".into(),
IrOp::LoopRestartIfFalse => "loop-restart-if-false".into(),
// -- Flat forward branches --
IrOp::Block(l) => format!("block L{l}"),
IrOp::BranchIfFalse(l) => format!("branch-if-false L{l}"),
IrOp::EndBlock(l) => format!("end-block L{l}"),
// -- Return stack --
IrOp::ToR => ">r".into(),
IrOp::FromR => "r>".into(),
IrOp::RFetch => "r@".into(),
IrOp::LoopJ => "j".into(),
// -- Forth locals --
IrOp::ForthLocalGet(n) => format!("local@ {n}"),
IrOp::ForthLocalSet(n) => format!("local! {n}"),
IrOp::ForthFLocalGet(n) => format!("flocal@ {n}"),
IrOp::ForthFLocalSet(n) => format!("flocal! {n}"),
// -- I/O --
IrOp::Emit => "emit".into(),
IrOp::Dot => ".".into(),
IrOp::Cr => "cr".into(),
IrOp::Type => "type".into(),
// -- System --
IrOp::Execute => "execute".into(),
IrOp::SpFetch => "sp@".into(),
IrOp::RpFetch => "rp@".into(),
// -- Float stack --
IrOp::FDup => "fdup".into(),
IrOp::FDrop => "fdrop".into(),
IrOp::FSwap => "fswap".into(),
IrOp::FOver => "fover".into(),
// -- Float arithmetic --
IrOp::FAdd => "fadd".into(),
IrOp::FSub => "fsub".into(),
IrOp::FMul => "fmul".into(),
IrOp::FDiv => "fdiv".into(),
IrOp::FNegate => "fnegate".into(),
IrOp::FAbs => "fabs".into(),
IrOp::FSqrt => "fsqrt".into(),
IrOp::FMin => "fmin".into(),
IrOp::FMax => "fmax".into(),
IrOp::FFloor => "ffloor".into(),
IrOp::FRound => "fround".into(),
// -- Float comparisons --
IrOp::FZeroEq => "f0=".into(),
IrOp::FZeroLt => "f0<".into(),
IrOp::FEq => "f=".into(),
IrOp::FLt => "f<".into(),
// -- Float memory --
IrOp::FetchFloat => "f@".into(),
IrOp::StoreFloat => "f!".into(),
// -- Conversions --
IrOp::StoF => "s>f".into(),
IrOp::FtoS => "f>s".into(),
};
line(out, depth, &simple);
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn simple_ops_one_per_line() {
let out = format_ir(&[IrOp::Dup, IrOp::Mul, IrOp::PushI32(7)]);
assert_eq!(out, "dup\nmul\npush 7\n");
}
#[test]
fn call_resolves_via_resolver() {
let ops = [IrOp::Call(WordId(12)), IrOp::TailCall(WordId(13))];
assert_eq!(format_ir(&ops), "call #12\ntail-call #13\n");
let named = format_ir_with(&ops, &|id| (id.0 == 12).then(|| "SQ".to_string()));
assert_eq!(named, "call SQ\ntail-call #13\n");
}
#[test]
fn nested_if_inside_do_loop_indents() {
let ops = [IrOp::DoLoop {
body: vec![
IrOp::Dup,
IrOp::If {
then_body: vec![IrOp::Dup, IrOp::Mul],
else_body: Some(vec![IrOp::Drop]),
},
],
is_plus_loop: false,
}];
let expected = "do\n dup\n if\n dup\n mul\n else\n drop\n then\nloop\n";
assert_eq!(format_ir(&ops), expected);
}
#[test]
fn while_loops_and_flat_branches() {
let ops = [
IrOp::BeginWhileRepeat {
test: vec![IrOp::Dup],
body: vec![IrOp::PushI32(1), IrOp::Sub],
},
IrOp::Block(3),
IrOp::BranchIfFalse(3),
IrOp::EndBlock(3),
];
let expected = "begin\n dup\nwhile\n push 1\n sub\nrepeat\nblock L3\nbranch-if-false L3\nend-block L3\n";
assert_eq!(format_ir(&ops), expected);
}
#[test]
fn every_simple_variant_renders() {
// One of each non-structured op; count of output lines must match.
let ops = vec![
IrOp::PushI32(1),
IrOp::PushI64(2),
IrOp::PushF64(1.5),
IrOp::Drop,
IrOp::Dup,
IrOp::Swap,
IrOp::Over,
IrOp::Rot,
IrOp::Nip,
IrOp::Tuck,
IrOp::TwoDup,
IrOp::TwoDrop,
IrOp::Add,
IrOp::Sub,
IrOp::Mul,
IrOp::DivMod,
IrOp::Negate,
IrOp::Abs,
IrOp::Eq,
IrOp::NotEq,
IrOp::Lt,
IrOp::Gt,
IrOp::LtUnsigned,
IrOp::ZeroEq,
IrOp::ZeroLt,
IrOp::And,
IrOp::Or,
IrOp::Xor,
IrOp::Invert,
IrOp::Lshift,
IrOp::Rshift,
IrOp::ArithRshift,
IrOp::Fetch,
IrOp::Store,
IrOp::CFetch,
IrOp::CStore,
IrOp::PlusStore,
IrOp::Call(WordId(1)),
IrOp::TailCall(WordId(2)),
IrOp::Exit,
IrOp::LoopRestartIfFalse,
IrOp::Block(1),
IrOp::BranchIfFalse(1),
IrOp::EndBlock(1),
IrOp::ToR,
IrOp::FromR,
IrOp::RFetch,
IrOp::LoopJ,
IrOp::ForthLocalGet(0),
IrOp::ForthLocalSet(0),
IrOp::ForthFLocalGet(0),
IrOp::ForthFLocalSet(0),
IrOp::Emit,
IrOp::Dot,
IrOp::Cr,
IrOp::Type,
IrOp::Execute,
IrOp::SpFetch,
IrOp::RpFetch,
IrOp::FDup,
IrOp::FDrop,
IrOp::FSwap,
IrOp::FOver,
IrOp::FAdd,
IrOp::FSub,
IrOp::FMul,
IrOp::FDiv,
IrOp::FNegate,
IrOp::FAbs,
IrOp::FSqrt,
IrOp::FMin,
IrOp::FMax,
IrOp::FFloor,
IrOp::FRound,
IrOp::FZeroEq,
IrOp::FZeroLt,
IrOp::FEq,
IrOp::FLt,
IrOp::FetchFloat,
IrOp::StoreFloat,
IrOp::StoF,
IrOp::FtoS,
];
let out = format_ir(&ops);
assert_eq!(out.lines().count(), ops.len());
// Every line non-empty, no accidental blank rendering.
assert!(out.lines().all(|l| !l.trim().is_empty()));
}
}
File diff suppressed because it is too large Load Diff
+183 -57
View File
@@ -1,8 +1,10 @@
#![allow(dead_code)]
//! Cross-engine comparison tests: WAFER vs gforth.
//! Cross-engine comparison tests: WAFER vs gforth (and `SwiftForth` for perf).
//!
//! Validates that WAFER produces identical output to gforth for standard
//! Forth programs, and benchmarks performance of both engines.
//! Forth programs, and benchmarks performance of the engines. `SwiftForth`
//! (`sf64`, native-code commercial compiler) joins the performance report
//! as an upper-bound reference when installed.
//!
//! WAFER-only correctness: `cargo test -p wafer-core --test comparison`
//! Full comparison + perf: `cargo test -p wafer-core --test comparison -- --nocapture --ignored`
@@ -26,8 +28,7 @@ fn probe_gforth(candidate: &str) -> bool {
.arg("-e")
.arg("bye")
.output()
.map(|o| o.status.success())
.unwrap_or(false)
.is_ok_and(|o| o.status.success())
}
fn find_gforth() -> Option<&'static str> {
@@ -64,6 +65,48 @@ fn find_gforth_fast() -> Option<&'static str> {
.as_deref()
}
// -----------------------------------------------------------------------
// SwiftForth (sf64) discovery (cached)
// -----------------------------------------------------------------------
static SF64_PATH: OnceLock<Option<String>> = OnceLock::new();
/// Probe sf64 by piping `bye` via stdin — sf64 has no `-e` flag; it takes
/// Forth source from stdin or as bare command-line arguments.
fn probe_sf64(candidate: &str) -> bool {
run_via_stdin(candidate, "bye\n").is_some_and(|o| o.status.success())
}
fn find_sf64() -> Option<&'static str> {
SF64_PATH
.get_or_init(|| {
for candidate in &["/Applications/ForthInc/SwiftForth/bin/macos/sf64", "sf64"] {
if probe_sf64(candidate) {
return Some(candidate.to_string());
}
}
None
})
.as_deref()
}
/// Spawn `binary`, write `input` to its stdin, and collect the output.
fn run_via_stdin(binary: &str, input: &str) -> Option<std::process::Output> {
Command::new(binary)
// Perf lanes measure unguarded code (only the wafer binary reads this)
.env("WAFER_STACK_GUARDS", "0")
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()
.and_then(|mut child| {
use std::io::Write;
child.stdin.take().unwrap().write_all(input.as_bytes())?;
child.wait_with_output()
})
.ok()
}
// -----------------------------------------------------------------------
// Engine runners
// -----------------------------------------------------------------------
@@ -581,6 +624,81 @@ fn compare_all_programs() {
);
}
// -----------------------------------------------------------------------
// Cross-engine behavioral comparison (requires SwiftForth sf64) -- WS-003
// -----------------------------------------------------------------------
/// Run Forth code through `SwiftForth`. Piped sf64 is quiet (no banner, no
/// `ok` echo), truncates input lines at ~256 chars, and exits 243 after an
/// error, so statements are fed one per line with a final `bye`.
fn run_sf64_code(sf64: &str, code: &str) -> Option<EngineResult> {
let mut input = String::new();
for line in code.lines() {
let t = line.trim();
if !t.is_empty() {
input.push_str(t);
input.push('\n');
}
}
input.push_str("bye\n");
let out = run_via_stdin(sf64, &input)?;
Some(EngineResult {
output: String::from_utf8_lossy(&out.stdout).to_string(),
success: out.status.success(),
})
}
/// Correctness lane against `SwiftForth`: the same program corpus as the
/// gforth comparison, sf64 as the oracle. Skips gracefully when sf64 is
/// not installed (CI/linux). Programs listed in `SF64_SKIP` use words or
/// output conventions `SwiftForth` does not share.
#[test]
#[ignore = "requires SwiftForth sf64 (run with -- --ignored)"]
fn compare_all_programs_sf64() {
// dot-quote: `."` outside a definition is a no-op in SwiftForth
// (compile-only); WAFER supports the interpret-mode extension.
const SF64_SKIP: &[&str] = &["dot-quote"];
let Some(sf64) = find_sf64() else {
eprintln!("SKIP: sf64 not found");
return;
};
let progs = programs();
let mut passed = 0;
let mut skipped = 0;
for prog in &progs {
if SF64_SKIP.contains(&prog.name) {
skipped += 1;
continue;
}
let wafer = run_wafer(prog.code);
assert!(wafer.success, "{}: WAFER execution failed", prog.name);
let Some(sf) = run_sf64_code(sf64, prog.code) else {
skipped += 1;
continue;
};
if !sf.success {
eprintln!(" WARN {}: sf64 execution failed, skipping", prog.name);
skipped += 1;
continue;
}
// SwiftForth prints numbers space-prefixed and echoes piped input
// lines, so byte-exact comparison is meaningless; compare the
// whitespace-token stream (the printed values and strings).
let wafer_tokens: Vec<&str> = wafer.output.split_whitespace().collect();
let sf_tokens: Vec<&str> = sf.output.split_whitespace().collect();
assert_eq!(
wafer_tokens, sf_tokens,
"{}: output differs\n WAFER: {:?}\n sf64: {:?}",
prog.name, wafer.output, sf.output
);
passed += 1;
}
eprintln!(
"\nsf64 behavioral comparison: {passed} passed, {skipped} skipped (of {})",
progs.len()
);
}
// -----------------------------------------------------------------------
// Performance comparison (requires gforth)
// -----------------------------------------------------------------------
@@ -699,31 +817,11 @@ fn measure_wafer_release(wafer: &str, bench: &PerfBenchmark) -> Option<u64> {
define = bench.define,
run = bench.run_code,
);
let output = Command::new(wafer)
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()
.and_then(|mut child| {
use std::io::Write;
child.stdin.take().unwrap().write_all(code.as_bytes())?;
child.wait_with_output()
})
.ok()?;
let output = run_via_stdin(wafer, &code)?;
if !output.status.success() {
return None;
}
let stdout = String::from_utf8_lossy(&output.stdout);
let mut times: Vec<u64> = stdout
.trim()
.lines()
.filter_map(|l| l.trim().parse::<u64>().ok())
.collect();
times.sort();
if times.is_empty() {
return None;
}
Some(times[times.len() / 2])
median_printed_time(&output.stdout)
}
/// Measure WAFER execution time after CONSOLIDATE (direct calls between all words).
@@ -735,21 +833,17 @@ fn measure_wafer_consolidated(wafer: &str, bench: &PerfBenchmark) -> Option<u64>
define = bench.define,
run = bench.run_code,
);
let output = Command::new(wafer)
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()
.and_then(|mut child| {
use std::io::Write;
child.stdin.take().unwrap().write_all(code.as_bytes())?;
child.wait_with_output()
})
.ok()?;
let output = run_via_stdin(wafer, &code)?;
if !output.status.success() {
return None;
}
let stdout = String::from_utf8_lossy(&output.stdout);
median_printed_time(&output.stdout)
}
/// Parse the microsecond values printed by TIMED-BENCH (one per line) and
/// return the median.
fn median_printed_time(stdout: &[u8]) -> Option<u64> {
let stdout = String::from_utf8_lossy(stdout);
let mut times: Vec<u64> = stdout
.trim()
.lines()
@@ -779,18 +873,28 @@ fn measure_gforth(gforth: &str, bench: &PerfBenchmark) -> Option<u64> {
if !output.status.success() {
return None;
}
let stdout = String::from_utf8_lossy(&output.stdout);
// Parse the 3 timing values and take the median
let mut times: Vec<u64> = stdout
.trim()
.lines()
.filter_map(|l| l.trim().parse::<u64>().ok())
.collect();
times.sort();
if times.is_empty() {
median_printed_time(&output.stdout)
}
/// Measure `SwiftForth` (`sf64`) execution time using Forth-level `ucounter`
/// (double-cell microsecond counter; `2swap d- drop` yields elapsed us —
/// the same wrapper shape as gforth's `utime`). Timing excludes startup.
/// sf64 has no `-e` flag, so the program is piped via stdin — one statement
/// per line, because sf64 truncates input lines at ~256 chars.
/// Returns microseconds, or None if sf64 is unavailable or fails.
fn measure_sf64(sf64: &str, bench: &PerfBenchmark) -> Option<u64> {
let code = format!(
"{define}\n{run}\n\
: TIMED-BENCH ucounter {run} ucounter 2swap d- drop . cr ;\n\
TIMED-BENCH\nTIMED-BENCH\nTIMED-BENCH\nbye\n",
define = bench.define,
run = bench.run_code,
);
let output = run_via_stdin(sf64, &code)?;
if !output.status.success() {
return None;
}
Some(times[times.len() / 2])
median_printed_time(&output.stdout)
}
#[test]
@@ -831,18 +935,31 @@ fn performance_report() {
);
}
let sep = "=".repeat(80);
let thin = "-".repeat(80);
let sf64 = find_sf64();
if sf64.is_none() {
eprintln!("NOTE: sf64 (SwiftForth) not found — column skipped");
}
let sep = "=".repeat(100);
let thin = "-".repeat(100);
println!("\n{sep}");
println!(" WAFER vs Gforth Performance Comparison (release mode)");
println!(" WAFER vs Gforth vs SwiftForth Performance Comparison (release mode)");
println!("{sep}\n");
println!(
"{:<22} {:>10} {:>10} {:>10} {:>10} {:>10} {:>10}",
"Benchmark", "WAFER", "CONSOL", "gforth", "gf-fast", "WAFER/gf", "limit"
"{:<22} {:>9} {:>9} {:>9} {:>9} {:>9} {:>9} {:>9} {:>9}",
"Benchmark",
"WAFER",
"CONSOL",
"gforth",
"gf-fast",
"sf64",
"WAFER/gf",
"WAFER/sf",
"limit"
);
println!(
"{:<22} {:>10} {:>10} {:>10} {:>10} {:>10} {:>10}",
"", "(us)", "(us)", "(us)", "(us)", "", ""
"{:<22} {:>9} {:>9} {:>9} {:>9} {:>9} {:>9} {:>9} {:>9}",
"", "(us)", "(us)", "(us)", "(us)", "(us)", "", "", ""
);
println!("{thin}");
@@ -857,9 +974,11 @@ fn performance_report() {
.unwrap_or(0);
let gf = gforth.and_then(|g| measure_gforth(g, bench));
let gf_fast = gforth_fast.and_then(|g| measure_gforth(g, bench));
let sf = sf64.and_then(|s| measure_sf64(s, bench));
let gf_str = gf.map_or_else(|| "-".to_string(), |v| format!("{v}"));
let gf_fast_str = gf_fast.map_or_else(|| "-".to_string(), |v| format!("{v}"));
let sf_str = sf.map_or_else(|| "-".to_string(), |v| format!("{v}"));
let best_wafer = if consol > 0 && consol < wafer {
consol
} else {
@@ -873,11 +992,15 @@ fn performance_report() {
}
});
let ratio = ratio_val.map_or_else(|| "-".to_string(), |r| format!("{r:.2}x"));
let sf_ratio = sf.filter(|&s| s > 0).map_or_else(
|| "-".to_string(),
|s| format!("{:.2}x", best_wafer as f64 / s as f64),
);
let limit_str = format!("{:.2}x", bench.max_ratio);
println!(
"{:<22} {:>10} {:>10} {:>10} {:>10} {:>10} {:>10}",
bench.name, wafer, consol, gf_str, gf_fast_str, ratio, limit_str
"{:<22} {:>9} {:>9} {:>9} {:>9} {:>9} {:>9} {:>9} {:>9}",
bench.name, wafer, consol, gf_str, gf_fast_str, sf_str, ratio, sf_ratio, limit_str
);
// Check regression limits
@@ -900,6 +1023,9 @@ fn performance_report() {
println!("{thin}");
println!(" WAFER = all optimizations, CONSOL = after CONSOLIDATE");
println!(" WAFER/gf = best(WAFER,CONSOL) vs gforth, < 1.0 means WAFER faster");
println!(
" WAFER/sf = best(WAFER,CONSOL) vs SwiftForth sf64 (native code; informational, no limit)"
);
println!("{sep}\n");
if !regressions.is_empty() {
+6 -1
View File
@@ -105,8 +105,13 @@ fn expected_load_failures(path: &str) -> u32 {
// TRAVERSE-WORDLIST / NAME>COMPILE / NAME>INTERPRET blocks leak as
// unknown-word errors. Fix the SOURCE/`>IN` interaction with
// line-mode input and drop this to 0.
//
// The 38th: line 368 `R> DROP TRUE` runs interpreted (its enclosing
// definition aborted on the missing NAME?), and the bare `R>` used
// to underflow the return stack silently; stack guards now report
// it as "Return stack underflow (throw -6)".
if path.ends_with("/toolstest.fth") {
return 37;
return 38;
}
0
}
+8 -4
View File
@@ -6,8 +6,9 @@ use send_wrapper::SendWrapper;
use wasm_bindgen::prelude::*;
use wafer_core::config::WaferConfig;
use wafer_core::memory::{CELL_SIZE, PAD_BASE, PAD_SIZE};
use wafer_core::memory::{CELL_SIZE, PAD_BASE, PAD_SIZE, SYSVAR_BASE_VAR};
use wafer_core::outer::ForthVM;
use wafer_core::runtime::Runtime;
use wafer_core::runtime::{HostAccess, HostFn};
use crate::runtime_web::WebRuntime;
@@ -53,9 +54,12 @@ impl WaferRepl {
/// Get the current number base (10 = decimal, 16 = hex).
pub fn base(&mut self) -> u32 {
// BASE is stored at SYSVAR_BASE_VAR in WASM memory
self.vm.take_output(); // no-op side effect; just return base
10 // TODO: read from memory once we have a getter
self.vm.runtime_mut().mem_read_i32(SYSVAR_BASE_VAR) as u32
}
/// Names of all user-facing words (visible, non-internal), newest first.
pub fn words(&self) -> Vec<String> {
self.vm.word_names()
}
/// Reset the VM to initial state.
+42 -23
View File
@@ -1,8 +1,11 @@
import init, { WaferRepl } from './pkg/wafer_web.js';
let repl = null;
const history = [];
let historyIdx = -1;
const HISTORY_KEY = 'wafer-history';
const HISTORY_MAX = 200;
const history = JSON.parse(localStorage.getItem(HISTORY_KEY) || '[]');
let historyIdx = history.length;
let builtinWords = null;
const WORD_CATEGORIES = {
'Stack': 'DUP DROP SWAP OVER ROT NIP TUCK 2DUP 2DROP 2SWAP 2OVER PICK ROLL DEPTH .S'.split(' '),
@@ -39,10 +42,12 @@ function updateStack() {
if (!repl) return;
try {
const stack = repl.data_stack();
const base = repl.base();
const suffix = base !== 10 ? ` [base ${base}]` : '';
if (stack.length === 0) {
stackBar.textContent = 'Stack: (empty)';
stackBar.textContent = `Stack: (empty)${suffix}`;
} else {
stackBar.textContent = `Stack <${stack.length}> ${stack.join(' ')}`;
stackBar.textContent = `Stack <${stack.length}> ${stack.join(' ')}${suffix}`;
}
} catch {
stackBar.textContent = 'Stack: (error)';
@@ -50,19 +55,25 @@ function updateStack() {
}
function updateUserWords() {
const cat = document.getElementById('cat-user');
if (!cat) return;
// We'll track user words by checking what the REPL evaluates
// For now, just show the category
const list = document.getElementById('user-word-list');
if (!list || !repl || !builtinWords) return;
list.innerHTML = '';
for (const w of repl.words()) {
if (!builtinWords.has(w)) list.appendChild(wordChip(w));
}
}
function evaluate(line) {
function evaluate(line, record = true) {
if (!repl) return;
const trimmed = line.trim();
if (!trimmed) return;
// Add to history
history.push(trimmed);
// Add to history (user-typed lines only; skip consecutive duplicates)
if (record && history[history.length - 1] !== trimmed) {
history.push(trimmed);
if (history.length > HISTORY_MAX) history.splice(0, history.length - HISTORY_MAX);
localStorage.setItem(HISTORY_KEY, JSON.stringify(history));
}
historyIdx = history.length;
try {
@@ -83,6 +94,7 @@ function evaluate(line) {
updatePrompt();
updateStack();
updateUserWords();
}
// Input handling
@@ -116,6 +128,18 @@ document.getElementById('btn-toggle-words').addEventListener('click', () => {
document.getElementById('word-panel').classList.toggle('collapsed');
});
function wordChip(w) {
const chip = document.createElement('span');
chip.className = 'word-chip';
chip.textContent = w;
chip.title = w;
chip.addEventListener('click', () => {
input.value += (input.value.length > 0 ? ' ' : '') + w;
input.focus();
});
return chip;
}
function buildWordPanel() {
const container = document.getElementById('word-categories');
container.innerHTML = '';
@@ -129,15 +153,7 @@ function buildWordPanel() {
const list = document.createElement('div');
list.className = 'word-list';
for (const w of words) {
const chip = document.createElement('span');
chip.className = 'word-chip';
chip.textContent = w;
chip.title = w;
chip.addEventListener('click', () => {
input.value += (input.value.length > 0 ? ' ' : '') + w;
input.focus();
});
list.appendChild(chip);
list.appendChild(wordChip(w));
}
cat.appendChild(list);
container.appendChild(cat);
@@ -179,7 +195,7 @@ document.getElementById('btn-run-init').addEventListener('click', () => {
if (code.trim()) {
// Run each line separately
for (const line of code.split('\n')) {
if (line.trim()) evaluate(line);
if (line.trim()) evaluate(line, false);
}
}
localStorage.setItem('wafer-init-code', code);
@@ -214,6 +230,7 @@ document.getElementById('btn-reset').addEventListener('click', () => {
appendLine('WAFER reset.', 'line-ok');
updatePrompt();
updateStack();
updateUserWords();
} catch (e) {
appendLine(`Reset error: ${e.message}`, 'line-error');
}
@@ -225,6 +242,8 @@ async function boot() {
try {
await init();
repl = new WaferRepl();
// Everything defined at boot is "builtin"; later definitions are user words
builtinWords = new Set(repl.words());
output.innerHTML = '';
appendLine('WAFER — WebAssembly Forth Engine in Rust', 'line-output');
appendLine(`Type Forth at the > prompt. Press ? for help.`, 'line-output');
@@ -241,7 +260,7 @@ async function boot() {
const initCode = document.getElementById('init-code').value;
if (initCode.trim()) {
for (const line of initCode.split('\n')) {
if (line.trim()) evaluate(line);
if (line.trim()) evaluate(line, false);
}
localStorage.setItem('wafer-init-code', initCode);
}
@@ -252,7 +271,7 @@ async function boot() {
const code = atob(location.hash.slice(1));
document.getElementById('init-code').value = code;
for (const line of code.split('\n')) {
if (line.trim()) evaluate(line);
if (line.trim()) evaluate(line, false);
}
} catch { /* ignore bad hash */ }
}
+230
View File
@@ -0,0 +1,230 @@
# Plan: SEE / SEE-IR / HELP — Introspection Trio
Status: implemented 2026-08-05 (all phases; HELP covers every word in a fresh VM, enforced by test)
Scope: `SEE` (source-level decompile), `SEE-IR` (optimized-IR dump), `HELP` (per-word docs), shared lookup infrastructure.
Each phase is self-contained and executable in a fresh context. Execute in order; every phase leaves the tree green (`cargo test --workspace` passes).
---
## Phase 0 — Consolidated Findings (read this first, do not re-derive)
All references verified at commit `e31407a` (branch `usability`).
### The template to copy: WORDS
`WORDS` is a **host primitive whose body runs Rust-side via the `pending_define` mechanism**. This is the exact pattern for SEE/SEE-IR/HELP because it gives a real dictionary entry (→ findable by `'`, listed by `WORDS`, tab-completable in the CLI via `crates/cli/src/main.rs:452-455`, exposed to web palette via `crates/web/src/lib.rs:61`) while the implementation can still call `next_token()` and write `self.output`.
- Registration: `register_words()` at `crates/core/src/outer.rs:6301-6310` — host fn pushes code `40` into `pending_define`, called from `register_primitives()` at `outer.rs:2991` under the `// -- Programming-Tools word set --` header.
- Dispatch: `handle_pending_define()` arm at `outer.rs:5328`: `40 => self.do_words(),`.
- Body: `do_words()` at `outer.rs:6045-6074`. Note `outer.rs:6050-6052`: it reads an optional same-line argument with `self.next_token()` **gated on `self.state == 0`** — SEE must copy this gate.
- **Used `pending_define` codes: 112, 20, 21, 25, 33, 40.** Free: **41 (SEE), 42 (SEE-IR), 43 (HELP)**. Legend comment at `outer.rs:232-233` must be extended.
### Allowed APIs (verified signatures)
| API | Location | Notes |
|---|---|---|
| `Dictionary::find(&self, name: &str) -> Option<(u32, WordId, bool)>` | `dictionary.rs:182` | `(word_addr, WordId, is_immediate)`, case-insensitive |
| `Dictionary::word_name(word_addr)` / `code_field(word_addr)` / `read_link` / `latest()` | `dictionary.rs:375/392/287/282` | manual entry walk |
| `flags::IMMEDIATE = 0x80`, `HIDDEN = 0x40`, `INTERNAL = 0x20` | `dictionary.rs:16-27` | raw flags byte = `dict.memory()[(word_addr+4) as usize]` — no getter exists |
| `ir_bodies: HashMap<WordId, Vec<IrOp>>` | `outer.rs:256` | **post-optimization** IR; populated for colon words AND all defining-word products AND IR primitives (see kind table below) |
| `host_word_names: HashMap<WordId, String>` | `outer.rs:214` | only populated by `register_host_primitive` (`outer.rs:2702-2703`) |
| `does_definitions: HashMap<WordId, DoesDefinition>` | `outer.rs:223` | DOES>-words |
| `output: Arc<Mutex<String>>` | `outer.rs:208` | ALL text output goes here; `HostAccess` has **no** emit method (`runtime.rs:17-60`) |
| `next_token()` | `outer.rs:690-704` | whitespace-delimited, advances `input_pos` |
| `register_host_primitive(name, immediate, func) -> anyhow::Result<WordId>` | `outer.rs:2686-2691` | public |
| `IrOp` enum, `#[derive(Debug, Clone, PartialEq)]` | `ir.rs:9-218` | **no `Display` impl exists anywhere in core** — formatter is net-new |
| `eval_output(input) -> String` test helper | `outer.rs:7566` | fresh VM per call; multi-eval tests build VM inline like `outer.rs:9077-9080` |
### Word-kind classification (SEE must distinguish these)
| Kind | Detectable via | SEE output strategy |
|---|---|---|
| Colon word / `:NONAME` | in `ir_bodies`, has captured source (Phase 3) | source (Phase 3) or IR (Phase 2) |
| IR primitive (`DUP`…) | in `ir_bodies`, no source | IR body + "primitive" tag |
| Host primitive (`.S`, `WORDS`…) | in `host_word_names` | `<built-in host word>` stub |
| CONSTANT / VARIABLE / VALUE / CREATE / DEFER / SYNONYM / BUFFER: / 2\*/F\* | in `ir_bodies` with recognizable shape (e.g. CONSTANT = `[PushI32(v)]`, insert sites: `outer.rs:3194/3226/3263/3309/3351/3388/3440/6517/6547/6590/7344`) | synthesized definition, e.g. `42 CONSTANT ANSWER` (Phase 3) |
| DOES>-defined word | key in `does_definitions` | show CREATE part + DOES> body IR |
| Interpreter special token (`:`, `;`, `VARIABLE`, `'`, `CHAR`…) | hardcoded matches `outer.rs:755-820`, `927-992`; several have **no dictionary entry at all** | `<compiler word, handled by the outer interpreter>` stub |
### Hard constraints
1. **Feature-free.** `outer.rs`, `ir.rs`, `dictionary.rs` compile without the `native` feature (`lib.rs:17-42`); web consumes core with `default-features = false` (`crates/web/Cargo.toml:15`). No `#[cfg(feature = "native")]` in any SEE code. Unit tests live in the existing `#[cfg(all(test, feature = "native"))]` module (`outer.rs:7553`) — that is fine and matches practice.
2. **`ir_bodies` stores post-optimization IR** (`finish_colon_def`: optimize at `outer.rs:2297`, insert at `outer.rs:2298`; inlining threshold 8 at `optimizer.rs:56`). `: FOO SQ SQ ;` shows `SQ`'s body inlined. This is a *feature* for SEE-IR (shows what the optimizer did) and the *reason* SEE needs separate source capture (Phase 3).
3. **Multi-line definitions**: compile state persists across `evaluate()` calls (`outer.rs:512-514` resets only `input_buffer`/`input_pos`); the driver (CLI `main.rs:411-416`) feeds lines. Source capture must accumulate across calls. On error, `evaluate()` wipes compile state (`outer.rs:523-535`) — capture state must be wiped there too.
4. **Error house style** (`outer.rs:1294/3400/4057` precedents): `anyhow::bail!("SEE: unknown word: {name}")`, `anyhow::bail!("SEE: expected word name")`.
5. **Compliance suite gives SEE zero coverage**`toolstest.fth:38-39` explicitly excludes it. All coverage is hand-written unit tests. Adding SEE cannot break `compliance_tools`.
6. **MARKER correctness**: any new per-word map (source text, docs) must be snapshotted/restored in `MarkerState` (`outer.rs:166-176`, snapshot `outer.rs:3462-3480`, restore `outer.rs:3484-3506`), mirroring how `ir_bodies` is handled there.
### Anti-patterns (verified NOT to exist — do not invent)
- `HostAccess::emit(...)` / any output method on `HostAccess` — does not exist; capture `Arc::clone(&self.output)` instead.
- `Display for IrOp` — does not exist; write the formatter.
- A dictionary "entry struct" or kind tag — does not exist; classify via the VM-side maps above.
- `Dictionary::flags(addr)` getter — does not exist; read the raw byte.
- Refill-on-demand for a missing SEE argument — `REFILL`/`ACCEPT` are hardcoded to fail (`outer.rs:5824-5852`); `SEE` at end of line is an error, same as `'` (`outer.rs:4051-4053`).
---
## Phase 1 — IR pretty-printer (pure function, no VM changes)
**Goal:** a feature-free formatter turning `&[IrOp]` into readable, indented text. Foundation for SEE-IR and the SEE fallback path.
**What to implement:**
1. New module `crates/core/src/see.rs`, registered unconditionally in `lib.rs` next to `pub mod outer;` (`lib.rs:30`). Public API:
```rust
/// Format an IR body as indented, one-op-per-line text.
pub fn format_ir(ops: &[IrOp]) -> String
```
2. Exhaustive `match` over every `IrOp` variant (full list at `ir.rs:10-218`) — **no wildcard arm**, so adding a variant later forces a formatter update at compile time.
3. Simple ops print as their Forth-ish name plus payload: `PushI32(7)``push 7`, `Call(WordId(12))``call #12`, `TailCall``tail-call #12`. Resolve `#12` to a word name at a higher level (Phase 2) — `format_ir` itself stays name-agnostic, but takes an optional resolver to keep it pure:
```rust
pub fn format_ir_with(ops: &[IrOp], resolve: &dyn Fn(WordId) -> Option<String>) -> String
```
(`format_ir` delegates with a `|_| None` resolver.)
4. The six nested variants (`If`, `DoLoop`, `BeginUntil`, `BeginAgain`, `BeginWhileRepeat` at `ir.rs:78-99`, `BeginDoubleWhileRepeat` at `ir.rs:105-111`) print as Forth control words with 2-space indented bodies:
```
if
dup
mul
else
drop
then
```
5. Flat branch ops (`Block`/`BranchIfFalse`/`EndBlock`, `ir.rs:118-124`) print literally (`block L3` etc.) — they have no clean Forth surface syntax; do not attempt reconstruction.
**Verification checklist:**
- [ ] Unit tests in `see.rs` (plain `#[cfg(test)]`, NOT feature-gated — the module has no runtime dependency): nested `If` inside `DoLoop` indents correctly; every-variant smoke test via a `Vec` containing one of each simple op.
- [ ] `cargo check -p wafer-core --no-default-features` passes (proves feature-freedom).
- [ ] `cargo test --workspace` green; `cargo fmt --all` + `cargo clippy --workspace` clean.
**Anti-pattern guards:** no `impl Display for IrOp` (keep the formatter in `see.rs`, IrOp is data); no wildcard match arm; no `#[cfg(feature = "native")]`.
---## Phase 2 — SEE-IR word
**Goal:** `SEE-IR name` prints the stored post-optimization IR for any word — the optimizer-debugging view. Ship this before source-SEE: it is nearly free and immediately useful.
**What to implement:**
1. Copy the WORDS registration pattern verbatim (`outer.rs:6301-6310`): `register_see_ir()` pushes pending code **42**; register in `register_primitives()` next to `self.register_words()?` (`outer.rs:2991`). Extend the legend comment at `outer.rs:232-233`.
2. Dispatch arm in `handle_pending_define()` next to `outer.rs:5328`: `42 => self.do_see_ir(),`.
3. `do_see_ir()` (place near `do_words()`, `outer.rs:6045`):
- Parse name: `let Some(name) = self.next_token() else { bail!("SEE-IR: expected word name") }`**no** interpret-mode gate here (unlike WORDS' optional filter, the argument is mandatory; compile-mode `SEE-IR` may simply also parse — matches `'`).
- Lookup: `self.dictionary.find(&name)` → else `bail!("SEE-IR: unknown word: {name}")`.
- Classify per the Phase 0 kind table, in this order: `ir_bodies` hit → header line + `see::format_ir_with(...)` with a resolver that maps `WordId` → name (build once from a dictionary walk: `latest()`/`read_link`/`word_name`/`code_field`, `dictionary.rs:282/287/375/392`); `host_word_names` hit → `SEE-IR: <name> is a built-in host word`; neither → `SEE-IR: <name> has no IR body`.
- Header line format: `\ <NAME> — <n> ops (optimized IR)`, plus ` immediate` when the find() flag is set, plus `does>` info when `does_definitions` has the id.
- Write everything into `self.output.lock().unwrap()`; end with `\n` (multi-line output convention from commit `2910884`).
4. Special-token names (`:`, `VARIABLE`, `'`, …): after dictionary miss, check a small const list of known interpreter tokens (source: match arms at `outer.rs:755-820`, `927-992`) and print `SEE-IR: <name> is handled directly by the outer interpreter` instead of erroring.
**Documentation references:** WORDS pattern `outer.rs:6301-6310`, `5328`, `6045-6074`; error style `outer.rs:4057`; output convention `outer.rs:6056-6073`.
**Verification checklist:**
- [ ] Tests (in `outer.rs` test module, `eval_output` style, cf. `outer.rs:9035-9041`):
- `: SQ DUP * ; SEE-IR SQ` output contains `dup` and `mul`;
- `: FOO SQ SQ ; SEE-IR FOO` shows the **inlined** body (contains two `mul`, no `call`) — locks in the "optimized view" semantics;
- `SEE-IR DUP` works (IR primitive); `SEE-IR WORDS` prints host-word stub; `SEE-IR NOSUCHWORD` errors with `SEE-IR: unknown word: NOSUCHWORD`; bare `SEE-IR` errors with `expected word name`;
- `SEE-IR :` prints the interpreter-token message.
- [ ] `IF`/`ELSE`/`THEN` and `DO LOOP` bodies render indented (one structured-word test).
- [ ] `cargo test --workspace` green; fmt + clippy clean; `cargo check -p wafer-core --no-default-features` passes.
**Anti-pattern guards:** do not print via a nonexistent `HostAccess` emit; do not gate name parsing on `state == 0` (mandatory arg, not optional filter); do not `THROW -13` (plain `bail!` matches TO/SYNONYM precedent).
---
## Phase 3 — Source capture + SEE
**Goal:** `SEE name` prints the original source text `: name … ;` for colon words, synthesized definitions for data words, graceful stubs otherwise. This is the user-facing SEE.
**What to implement:**
1. **Capture fields** on `ForthVM` (near `compiling_ir`, `outer.rs:204`):
```rust
compiling_source: String, // accumulated raw text of the definition in progress
source_capture_from: Option<usize>, // input_pos where capture started in the CURRENT buffer
word_sources: HashMap<WordId, String>,
```
2. **Capture protocol** (verbatim source, including comments and string literals — token-level reassembly would lose them):
- `start_colon_def()` (`outer.rs:2097`): set `source_capture_from` to the position where `:` began. `interpret_token()` receives the token already consumed, so record the position **before** dispatch: in the `evaluate()` loop (`outer.rs:518-521`), remember `pos_before = self.input_pos` minus token — simplest correct form: capture `token_start` inside `next_token()` (`outer.rs:690-704`) into a new field `last_token_start: usize` as it skips whitespace; `start_colon_def` then does `self.source_capture_from = Some(self.last_token_start)`.
- End of `evaluate()` (after the loop, `outer.rs:~536`): if still compiling and capture active, flush `input_buffer[from..]` + `'\n'` into `compiling_source`, reset `source_capture_from = Some(0)` so the next buffer continues capture from its start.
- `finish_colon_def()` (`outer.rs:2266`): flush `input_buffer[from..=pos of ';']`, store `word_sources.insert(word_id, normalized)`, clear capture state. Normalize only trailing whitespace; keep interior verbatim.
- Error path `outer.rs:523-535`: clear both capture fields alongside the existing compile-state wipe.
- `:NONAME` and quotations (`outer.rs:759-761, 773-778`): skip capture (no name to SEE) — guard on `compiling_name.is_some()`.
3. **MARKER integration**: add `word_sources` to `MarkerState` (`outer.rs:166-176`), snapshot (`outer.rs:3462-3480`) and restore (`outer.rs:3484-3506`) exactly as `ir_bodies` is handled there.
4. **SEE word**: pending code **41**, same registration/dispatch shape as Phase 2. `do_see()` resolution order:
1. `word_sources` hit → print stored source verbatim, append ` immediate` on its own line if flagged (cf. `set_immediate`, `dictionary.rs:404`).
2. Recognizable data-word IR shape (Phase 0 kind table) → synthesized one-liner. CONSTANT `[PushI32(v)]``<v> CONSTANT <NAME>`; VARIABLE → `VARIABLE <NAME> ( addr=<v> )`; VALUE `[PushI32(a), Fetch]``<cur> VALUE <NAME>` reading current value via `self.rt` memory read if cheap, else `VALUE <NAME>`; SYNONYM `[Call(id)]``SYNONYM <NAME> <OLD>`; DEFER → `DEFER <NAME>` plus current target name via `does`/pfa lookup when resolvable.
3. `ir_bodies` hit (primitive or pre-capture colon word) → `\ <NAME> is a primitive; IR:` + `format_ir_with` output (reuse Phase 1/2 machinery — SEE never dead-ends).
4. `host_word_names` hit → `<NAME> is a built-in host word`.
5. Interpreter-token list → `<NAME> is handled by the outer interpreter (compiler word)`.
6. Else → `bail!("SEE: unknown word: {name}")`.
5. **Boot words get sources for free**: `boot.fth` definitions flow through the same `evaluate()`/`finish_colon_def` path, so `SEE NIP` etc. shows real boot source. Verify, don't assume — one test below.
**Documentation references:** compile-state lifecycle `outer.rs:512-535`, `2097-2121`, `2266-2329`; multi-line REPL driver `main.rs:411-416`; MarkerState `outer.rs:166-176, 3462-3506`.
**Verification checklist:**
- [ ] `: SQ DUP * ; SEE SQ` prints `: SQ DUP * ;` (verbatim, one line).
- [ ] Multi-line: inline-VM test (pattern `outer.rs:9077-9080`): `evaluate(": TRI\")` then `evaluate(\" DUP DUP ;")`, then `SEE TRI` shows both lines.
- [ ] Comment survives: `: C ( n -- n ) 1+ ; SEE C` output contains `( n -- n )`.
- [ ] `42 CONSTANT A SEE A``42 CONSTANT A`; `VARIABLE V SEE V` → contains `VARIABLE V`.
- [ ] `SEE NIP` (boot word) prints a colon definition, not an IR dump.
- [ ] `SEE DUP` prints the primitive-IR fallback; `SEE WORDS` prints host stub; `SEE '` prints interpreter-token message; unknown word errors in house style.
- [ ] MARKER round-trip: define word, set marker, redefine, execute marker, `SEE` shows the original — plus existing marker tests still green.
- [ ] Immediate flag: `: I2 ; IMMEDIATE SEE I2` output contains `immediate`.
- [ ] Error path: force `unknown word` mid-definition, then define a fresh word — its captured source must not contain debris from the aborted definition.
- [ ] Full suite + fmt + clippy + `--no-default-features` check.
**Anti-pattern guards:** do not reconstruct source from tokens (loses comments/strings/spacing); do not capture into `word_sources` for `:NONAME`; do not forget the error-path wipe (`outer.rs:523-535`) — stale capture corrupts the next definition's source; `evaluate()` resets `input_pos` per call (`outer.rs:512-514`) so `source_capture_from` is per-buffer, never carried across calls uncleared.
---
## Phase 4 — HELP word + doc table
**Goal:** `HELP name` prints stack effect + one-line description; `HELP` alone prints usage. Shares lookup/classification with SEE.
**What to implement:**
1. New feature-free module `crates/core/src/wordhelp.rs`: a static table
```rust
/// (NAME, stack effect, one-line description)
pub const WORD_DOCS: &[(&str, &str, &str)] = &[
("DUP", "( x -- x x )", "Duplicate the top of the data stack."),
...
];
pub fn lookup(name: &str) -> Option<(&'static str, &'static str)> // case-insensitive
```
Seed from the Forth 2012 glossary (stack effects are standardized). Cover, in priority order: core + core-ext words WAFER implements, then tools/double/float sets. Incomplete coverage is acceptable and expected — `HELP` says `no help for <name> (word exists)` when the word is defined but undocumented, which doubles as the TODO list.
2. `HELP` word: pending code **43**, same registration/dispatch shape as Phase 2. Resolution: parse optional name (bare `HELP` → usage line `HELP <word> — also try: WORDS, SEE <word>, SEE-IR <word>`); table hit → print `NAME ( stack effect ) description`; miss but dictionary hit → `no help for <name>` + hint `try SEE <name>`; miss both → house-style unknown-word error.
3. Cross-wiring (the "as useful as possible" part):
- `SEE`/`SEE-IR` prepend the HELP line as a `\ ...` comment when the table has one.
- `HELP` appends ` immediate` / `built-in` / `defined in boot.fth or user code` classification reusing the Phase 2/3 classifier — factor that classifier into a shared `fn classify_word(&self, name) -> WordClass` when Phase 4 lands (do NOT pre-build it in Phase 2; extract once there are two users, per smallest-change rule).
4. User-defined words: optional docstring convention — if the captured source's first parenthesized comment looks like a stack effect (`( ... -- ... )`), `HELP` echoes it for user words. No new syntax, zero cost, rewards idiomatic Forth style.
**Verification checklist:**
- [ ] `HELP DUP` prints stack effect + description; `HELP dup` (lowercase) same.
- [ ] `HELP` alone prints usage; `HELP NOSUCH` errors house-style; `HELP MYWORD` for undocumented-but-defined word prints the `no help` + `SEE` hint.
- [ ] `: SQ ( n -- n^2 ) DUP * ; HELP SQ` echoes `( n -- n^2 )`.
- [ ] Table lint test: iterate `WORD_DOCS`, assert every documented name resolves in a booted VM's dictionary (catches typos/renames mechanically).
- [ ] Full suite + fmt + clippy + `--no-default-features`.
**Anti-pattern guards:** no doc strings threaded through `register_primitive` call sites (200+ call-site churn, bloats outer.rs — the side table is deliberate); no partial-coverage panic — missing docs degrade gracefully.
---
## Phase 5 — Final verification + docs
1. **Full gate:** `cargo fmt --all` && `cargo clippy --workspace` (zero warnings) && `cargo test --workspace` (expect baseline 431 unit + new SEE/SEE-IR/HELP tests, 1 benchmark, 11 compliance, 9 comparison — all green).
2. **Feature-freedom proof:** `cargo check -p wafer-core --no-default-features` and web build `cd crates/web && wasm-pack build --target web --dev --out-dir www/pkg`.
3. **Manual REPL pass** (CLI): `SEE SQ`, `SEE-IR FOO` with inlining, `HELP DUP`, multi-line definition then SEE, tab-complete `SE<tab>` — confirm multi-line output renders per commit `2910884` conventions (block output, ` ok` on own line).
4. **Web REPL smoke:** serve `crates/web/www`, run the same commands — output flows through `take_output()` (`web/src/lib.rs:38-43`), no web-side changes expected.
5. **Anti-pattern grep:** `grep -n "cfg(feature" crates/core/src/see.rs crates/core/src/wordhelp.rs` → empty; `grep -n "impl Display for IrOp" -r crates/core` → empty; `grep -rn "emit" crates/core/src/see.rs` → empty.
6. **Docs:** `docs/FORTH.md:95` already lists SEE under Programming-Tools — verify claim now true; add SEE/SEE-IR/HELP to README feature list if words are enumerated there; extend CLAUDE.md test-count line.
7. **Compliance untouched:** `cargo test -p wafer-core --test compliance` — must stay 11/11 (suite excludes SEE by design, `toolstest.fth:38-39`).
---
## Deliberate scope cuts (revisit later, not now)
- **`SEE-WASM`** (disassemble compiled module via `wasmprinter`): compiled bytes are likely dropped after instantiation; `codegen.rs` unexamined. Separate plan if wanted.
- **IR→Forth source reconstruction** for optimized bodies: lossy and misleading post-inlining; the source-capture path makes it unnecessary.
- **`LOCATE` / editor integration**: needs file/line provenance in the dictionary; out of scope.
- **Forth-side doc syntax (`:doc`)**: revisit after self-hosting work starts; the `( n -- n^2 )` echo in Phase 4 covers the 80% case with zero syntax.
+32 -10
View File
@@ -33,7 +33,10 @@ contexts:
- include: compare
- include: memory
- include: io
- include: pictured
- include: string_ops
- include: float
- include: tools
- include: dictionary
- include: exception
- include: parsing
@@ -95,27 +98,31 @@ contexts:
# Quotations (Core-Ext 6.2.0455): [: ... ;] compiles an anonymous word.
- match: '(?i)(?:^|(?<=\s))(\[:|;\]){{ident_break}}'
scope: keyword.other.definition.forth
- match: '(?i)(?:^|(?<=\s))(VARIABLE|2VARIABLE|CONSTANT|2CONSTANT|VALUE|CREATE|DEFER|MARKER|BUFFER:|FCONSTANT|FVARIABLE)(\s+)(\S+)?'
- match: '(?i)(?:^|(?<=\s))(VARIABLE|2VARIABLE|CONSTANT|2CONSTANT|VALUE|CREATE|DEFER|MARKER|REMEMBER|BUFFER:|FCONSTANT|FVARIABLE)(\s+)(\S+)?'
captures:
1: keyword.other.defining.forth
3: entity.name.constant.forth
- match: '(?i)(?:^|(?<=\s))(DOES>|IMMEDIATE|RECURSE|POSTPONE|COMPILE,|LITERAL|2LITERAL|FLITERAL|SLITERAL){{ident_break}}'
- match: '(?i)(?:^|(?<=\s))(DOES>|IMMEDIATE|RECURSE|POSTPONE|COMPILE,|LITERAL|2LITERAL|FLITERAL|SLITERAL|DEFER!|DEFER@){{ident_break}}'
scope: keyword.other.defining.forth
control:
- match: '(?i)(?:^|(?<=\s))(IF|THEN|ELSE|BEGIN|UNTIL|WHILE|REPEAT|AGAIN|DO|\?DO|LOOP|\+LOOP|LEAVE|UNLOOP|EXIT|CASE|OF|ENDOF|ENDCASE|QUIT){{ident_break}}'
scope: keyword.control.forth
# Conditional compilation (Tools-ext 15.6.2).
- match: '(?i)(?:^|(?<=\s))(\[IF\]|\[ELSE\]|\[THEN\]|\[DEFINED\]|\[UNDEFINED\]){{ident_break}}'
scope: keyword.control.conditional-compilation.forth
stack_ops:
- match: '(?i)(?:^|(?<=\s))(DUP|\?DUP|DROP|SWAP|OVER|ROT|-ROT|NIP|TUCK|PICK|ROLL|2DUP|2DROP|2SWAP|2OVER|2ROT|DEPTH|SP@){{ident_break}}'
scope: support.function.stack.forth
return_stack:
- match: '(?i)(?:^|(?<=\s))(>R|R>|R@|2>R|2R>|2R@|N>R|NR>|I|J|CS-PICK|CS-ROLL){{ident_break}}'
# RP@ / RDEPTH are WAFER extensions (gforth-style return-stack access).
- match: '(?i)(?:^|(?<=\s))(>R|R>|R@|2>R|2R>|2R@|N>R|NR>|I|J|CS-PICK|CS-ROLL|RP@|RDEPTH){{ident_break}}'
scope: support.function.return-stack.forth
arithmetic:
- match: '(?i)(?:^|(?<=\s))(\+|-|\*|/|MOD|/MOD|\*/|\*/MOD|NEGATE|ABS|MIN|MAX|1\+|1-|2\*|2/|M\*|M\+|M\*/|UM\*|UM/MOD|FM/MOD|SM/REM|S>D|D>S){{ident_break}}'
- match: '(?i)(?:^|(?<=\s))(\+|-|\*|/|MOD|/MOD|\*/|\*/MOD|NEGATE|ABS|MIN|MAX|1\+|1-|2\*|2/|M\*|M\+|M\*/|UM\*|UM/MOD|FM/MOD|SM/REM|S>D|D>S|D\+|D-|DNEGATE|DABS|DMAX|DMIN|D2\*|D2/){{ident_break}}'
scope: keyword.operator.arithmetic.forth
logic:
@@ -123,21 +130,36 @@ contexts:
scope: keyword.operator.logical.forth
compare:
- match: '(?i)(?:^|(?<=\s))(=|<>|<|>|<=|>=|U<|U>|0=|0<>|0<|0>){{ident_break}}'
- match: '(?i)(?:^|(?<=\s))(=|<>|<|>|<=|>=|U<|U>|0=|0<>|0<|0>|D<|D=|D0<|D0=|DU<|WITHIN){{ident_break}}'
scope: keyword.operator.comparison.forth
memory:
- match: '(?i)(?:^|(?<=\s))(@|!|C@|C!|\+!|2@|2!|ALLOT|HERE|ALIGN|ALIGNED|CELL\+|CELLS|CHAR\+|CHARS|UNUSED|MOVE|CMOVE|CMOVE>|FILL|ERASE|BLANK|ALLOCATE|FREE|RESIZE|PAD){{ident_break}}'
- match: '(?i)(?:^|(?<=\s))(@|!|C@|C!|\+!|2@|2!|C,|ALLOT|HERE|ALIGN|ALIGNED|CELL\+|CELLS|CHAR\+|CHARS|UNUSED|MOVE|CMOVE|CMOVE>|FILL|ERASE|BLANK|ALLOCATE|FREE|RESIZE|PAD){{ident_break}}'
scope: support.function.memory.forth
io:
- match: '(?i)(?:^|(?<=\s))(EMIT|CR|SPACE|SPACES|TYPE|\.|U\.|\.R|U\.R|D\.|D\.R|\?|KEY|KEY\?|PAGE|AT-XY|ACCEPT|EXPECT|\.S){{ident_break}}'
- match: '(?i)(?:^|(?<=\s))(EMIT|CR|SPACE|SPACES|TYPE|\.|U\.|\.R|U\.R|D\.|D\.R|\?|KEY|KEY\?|PAGE|AT-XY|ACCEPT|EXPECT|\.S|F\.S|\.RS){{ident_break}}'
scope: support.function.io.forth
# Pictured numeric output (6.1: <# # #S #> HOLD SIGN; HOLDS is Core-Ext).
pictured:
- match: '(?i)(?:^|(?<=\s))(<#|#>|#S|#|HOLD|HOLDS|SIGN){{ident_break}}'
scope: support.function.pictured.forth
# String word set (17.6).
string_ops:
- match: '(?i)(?:^|(?<=\s))(COUNT|COMPARE|-TRAILING|/STRING){{ident_break}}'
scope: support.function.string.forth
float:
- match: '(?i)(?:^|(?<=\s))(F\+|F-|F\*|F/|FNEGATE|FABS|FMAX|FMIN|FSQRT|FFLOOR|FROUND|FSINCOS|F=|F<|F0=|F0<|F~|FDUP|FDROP|FSWAP|FOVER|FROT|FNIP|FTUCK|FDEPTH|F@|F!|FE\.|FS\.|F\.|F>D|D>F|F>S|S>F|>FLOAT|REPRESENT|PRECISION|SET-PRECISION|FALIGNED|DFALIGNED|SFALIGNED|DF@|DF!|SF@|SF!){{ident_break}}'
- match: '(?i)(?:^|(?<=\s))(F\+|F-|F\*\*|F\*|F/|FNEGATE|FABS|FMAX|FMIN|FSQRT|FFLOOR|FROUND|FLOOR|FSINCOS|FSINH|FSIN|FCOSH|FCOS|FTANH|FTAN|FASINH|FASIN|FACOSH|FACOS|FATANH|FATAN2|FATAN|FEXPM1|FEXP|FLNP1|FLN|FLOG|FALOG|F=|F<|F0=|F0<|F~|FDUP|FDROP|FSWAP|FOVER|FROT|FNIP|FTUCK|FDEPTH|F@|F!|FE\.|FS\.|F\.|F>D|D>F|F>S|S>F|>FLOAT|REPRESENT|PRECISION|SET-PRECISION|FALIGN|FALIGNED|DFALIGN|DFALIGNED|SFALIGN|SFALIGNED|FLOAT\+|FLOATS|DFLOAT\+|DFLOATS|SFLOAT\+|SFLOATS|DF@|DF!|SF@|SF!){{ident_break}}'
scope: support.function.float.forth
# Interactive/debug tools (Tools word set + WAFER REPL additions).
tools:
- match: '(?i)(?:^|(?<=\s))(SEE-IR|SEE|DUMP|BYE|HELP){{ident_break}}'
scope: support.function.tools.forth
dictionary:
- match: "(?i)(?:^|(?<=\\s))('|\\[']|,|>BODY|FIND|WORDS|ONLY|ALSO|PREVIOUS|DEFINITIONS|FORTH|GET-ORDER|SET-ORDER|GET-CURRENT|SET-CURRENT|WORDLIST|SEARCH-WORDLIST|FORTH-WORDLIST|ENVIRONMENT\\?|EXECUTE){{ident_break}}"
scope: support.function.dictionary.forth
@@ -147,7 +169,7 @@ contexts:
scope: keyword.control.exception.forth
parsing:
- match: '(?i)(?:^|(?<=\s))(PARSE|PARSE-NAME|WORD|REFILL|EVALUATE|SOURCE|SOURCE-ID|>IN|BASE|STATE|>NUMBER|SEARCH|SUBSTITUTE|UNESCAPE|REPLACES|S){{ident_break}}'
- match: '(?i)(?:^|(?<=\s))(PARSE|PARSE-NAME|WORD|REFILL|EVALUATE|INCLUDE|INCLUDED|SOURCE|SOURCE-ID|>IN|BASE|DECIMAL|HEX|STATE|>NUMBER|SEARCH|SUBSTITUTE|UNESCAPE|REPLACES|S){{ident_break}}'
scope: support.function.parsing.forth
literals:
@@ -185,5 +207,5 @@ contexts:
wafer_extras:
# WAFER-specific extensions beyond the Forth 2012 standard.
# When the language grows new user-facing non-standard words, add them here.
- match: '(?i)(?:^|(?<=\s))(CONSOLIDATE|RANDOM|RND-SEED|UTIME|READ-PASSWORD){{ident_break}}'
- match: '(?i)(?:^|(?<=\s))(CONSOLIDATE|RANDOM|RND-SEED|UTIME|READ-PASSWORD|EMPTY|GILD){{ident_break}}'
scope: support.function.wafer-extra.forth