WAFER/plans/01-see-introspection.md

Plan: SEE / SEE-IR / HELP — Introspection Trio

Status: implemented 2026-08-05 (all phases; HELP covers every word in a fresh VM, enforced by test) Scope: SEE (source-level decompile), SEE-IR (optimized-IR dump), HELP (per-word docs), shared lookup infrastructure. Each phase is self-contained and executable in a fresh context. Execute in order; every phase leaves the tree green (cargo test --workspace passes).

---

Phase 0 — Consolidated Findings (read this first, do not re-derive)

All references verified at commit e31407a (branch usability).

The template to copy: WORDS

WORDS is a host primitive whose body runs Rust-side via the pending_define mechanism. This is the exact pattern for SEE/SEE-IR/HELP because it gives a real dictionary entry (→ findable by ', listed by WORDS, tab-completable in the CLI via crates/cli/src/main.rs:452-455, exposed to web palette via crates/web/src/lib.rs:61) while the implementation can still call next_token() and write self.output.

• Registration: register_words() at crates/core/src/outer.rs:6301-6310 — host fn pushes code 40 into pending_define, called from register_primitives() at outer.rs:2991 under the // -- Programming-Tools word set -- header.
• Dispatch: handle_pending_define() arm at outer.rs:5328: 40 => self.do_words(),.
• Body: do_words() at outer.rs:6045-6074. Note outer.rs:6050-6052: it reads an optional same-line argument with self.next_token() gated on self.state == 0 — SEE must copy this gate.
• Used pending_define codes: 1–12, 20, 21, 25, 33, 40. Free: 41 (SEE), 42 (SEE-IR), 43 (HELP). Legend comment at outer.rs:232-233 must be extended.

Allowed APIs (verified signatures)

API	Location	Notes
Dictionary::find(&self, name: &str) -> Option<(u32, WordId, bool)>	dictionary.rs:182	(word_addr, WordId, is_immediate), case-insensitive
Dictionary::word_name(word_addr) / code_field(word_addr) / read_link / latest()	dictionary.rs:375/392/287/282	manual entry walk
flags::IMMEDIATE = 0x80, HIDDEN = 0x40, INTERNAL = 0x20	dictionary.rs:16-27	raw flags byte = dict.memory()[(word_addr+4) as usize] — no getter exists
ir_bodies: HashMap<WordId, Vec<IrOp>>	outer.rs:256	post-optimization IR; populated for colon words AND all defining-word products AND IR primitives (see kind table below)
host_word_names: HashMap<WordId, String>	outer.rs:214	only populated by register_host_primitive (outer.rs:2702-2703)
does_definitions: HashMap<WordId, DoesDefinition>	outer.rs:223	DOES>-words
output: Arc<Mutex<String>>	outer.rs:208	ALL text output goes here; HostAccess has no emit method (runtime.rs:17-60)
next_token()	outer.rs:690-704	whitespace-delimited, advances input_pos
register_host_primitive(name, immediate, func) -> anyhow::Result<WordId>	outer.rs:2686-2691	public
IrOp enum, #[derive(Debug, Clone, PartialEq)]	ir.rs:9-218	no Display impl exists anywhere in core — formatter is net-new
eval_output(input) -> String test helper	outer.rs:7566	fresh VM per call; multi-eval tests build VM inline like outer.rs:9077-9080

Word-kind classification (SEE must distinguish these)

Kind	Detectable via	SEE output strategy
Colon word / :NONAME	in ir_bodies, has captured source (Phase 3)	source (Phase 3) or IR (Phase 2)
IR primitive (DUP…)	in ir_bodies, no source	IR body + "primitive" tag
Host primitive (.S, WORDS…)	in host_word_names	<built-in host word> stub
CONSTANT / VARIABLE / VALUE / CREATE / DEFER / SYNONYM / BUFFER: / 2*/F*	in ir_bodies with recognizable shape (e.g. CONSTANT = [PushI32(v)], insert sites: outer.rs:3194/3226/3263/3309/3351/3388/3440/6517/6547/6590/7344)	synthesized definition, e.g. 42 CONSTANT ANSWER (Phase 3)
DOES>-defined word	key in does_definitions	show CREATE part + DOES> body IR
Interpreter special token (:, ;, VARIABLE, ', CHAR…)	hardcoded matches outer.rs:755-820, 927-992; several have no dictionary entry at all	<compiler word, handled by the outer interpreter> stub

Hard constraints

1. Feature-free. outer.rs, ir.rs, dictionary.rs compile without the native feature (lib.rs:17-42); web consumes core with default-features = false (crates/web/Cargo.toml:15). No #[cfg(feature = "native")] in any SEE code. Unit tests live in the existing #[cfg(all(test, feature = "native"))] module (outer.rs:7553) — that is fine and matches practice.
2. ir_bodies stores post-optimization IR (finish_colon_def: optimize at outer.rs:2297, insert at outer.rs:2298; inlining threshold 8 at optimizer.rs:56). : FOO SQ SQ ; shows SQ's body inlined. This is a feature for SEE-IR (shows what the optimizer did) and the reason SEE needs separate source capture (Phase 3).
3. Multi-line definitions: compile state persists across evaluate() calls (outer.rs:512-514 resets only input_buffer/input_pos); the driver (CLI main.rs:411-416) feeds lines. Source capture must accumulate across calls. On error, evaluate() wipes compile state (outer.rs:523-535) — capture state must be wiped there too.
4. Error house style (outer.rs:1294/3400/4057 precedents): anyhow::bail!("SEE: unknown word: {name}"), anyhow::bail!("SEE: expected word name").
5. Compliance suite gives SEE zero coverage — toolstest.fth:38-39 explicitly excludes it. All coverage is hand-written unit tests. Adding SEE cannot break compliance_tools.
6. MARKER correctness: any new per-word map (source text, docs) must be snapshotted/restored in MarkerState (outer.rs:166-176, snapshot outer.rs:3462-3480, restore outer.rs:3484-3506), mirroring how ir_bodies is handled there.

Anti-patterns (verified NOT to exist — do not invent)

• HostAccess::emit(...) / any output method on HostAccess — does not exist; capture Arc::clone(&self.output) instead.
• Display for IrOp — does not exist; write the formatter.
• A dictionary "entry struct" or kind tag — does not exist; classify via the VM-side maps above.
• Dictionary::flags(addr) getter — does not exist; read the raw byte.
• Refill-on-demand for a missing SEE argument — REFILL/ACCEPT are hardcoded to fail (outer.rs:5824-5852); SEE at end of line is an error, same as ' (outer.rs:4051-4053).

---

Phase 1 — IR pretty-printer (pure function, no VM changes)

Goal: a feature-free formatter turning &[IrOp] into readable, indented text. Foundation for SEE-IR and the SEE fallback path.

What to implement:

1. New module crates/core/src/see.rs, registered unconditionally in lib.rs next to pub mod outer; (lib.rs:30). Public API:

   /// Format an IR body as indented, one-op-per-line text.
   pub fn format_ir(ops: &[IrOp]) -> String
   

2. Exhaustive match over every IrOp variant (full list at ir.rs:10-218) — no wildcard arm, so adding a variant later forces a formatter update at compile time.
3. Simple ops print as their Forth-ish name plus payload: PushI32(7) → push 7, Call(WordId(12)) → call #12, TailCall → tail-call #12. Resolve #12 to a word name at a higher level (Phase 2) — format_ir itself stays name-agnostic, but takes an optional resolver to keep it pure:

   pub fn format_ir_with(ops: &[IrOp], resolve: &dyn Fn(WordId) -> Option<String>) -> String
   

   (format_ir delegates with a |_| None resolver.)
4. The six nested variants (If, DoLoop, BeginUntil, BeginAgain, BeginWhileRepeat at ir.rs:78-99, BeginDoubleWhileRepeat at ir.rs:105-111) print as Forth control words with 2-space indented bodies:

   if
     dup
     mul
   else
     drop
   then
   

5. Flat branch ops (Block/BranchIfFalse/EndBlock, ir.rs:118-124) print literally (block L3 etc.) — they have no clean Forth surface syntax; do not attempt reconstruction.

Verification checklist:

• Unit tests in see.rs (plain #[cfg(test)], NOT feature-gated — the module has no runtime dependency): nested If inside DoLoop indents correctly; every-variant smoke test via a Vec containing one of each simple op.
• cargo check -p wafer-core --no-default-features passes (proves feature-freedom).
• cargo test --workspace green; cargo fmt --all + cargo clippy --workspace clean.

Anti-pattern guards: no impl Display for IrOp (keep the formatter in see.rs, IrOp is data); no wildcard match arm; no #[cfg(feature = "native")].

---## Phase 2 — SEE-IR word

Goal: SEE-IR name prints the stored post-optimization IR for any word — the optimizer-debugging view. Ship this before source-SEE: it is nearly free and immediately useful.

What to implement:

1. Copy the WORDS registration pattern verbatim (outer.rs:6301-6310): register_see_ir() pushes pending code 42; register in register_primitives() next to self.register_words()? (outer.rs:2991). Extend the legend comment at outer.rs:232-233.
2. Dispatch arm in handle_pending_define() next to outer.rs:5328: 42 => self.do_see_ir(),.
3. do_see_ir() (place near do_words(), outer.rs:6045):
   • Parse name: let Some(name) = self.next_token() else { bail!("SEE-IR: expected word name") } — no interpret-mode gate here (unlike WORDS' optional filter, the argument is mandatory; compile-mode SEE-IR may simply also parse — matches ').
   • Lookup: self.dictionary.find(&name) → else bail!("SEE-IR: unknown word: {name}").
   • Classify per the Phase 0 kind table, in this order: ir_bodies hit → header line + see::format_ir_with(...) with a resolver that maps WordId → name (build once from a dictionary walk: latest()/read_link/word_name/code_field, dictionary.rs:282/287/375/392); host_word_names hit → SEE-IR: <name> is a built-in host word; neither → SEE-IR: <name> has no IR body.
   • Header line format: \ <NAME> — <n> ops (optimized IR), plus immediate when the find() flag is set, plus does> info when does_definitions has the id.
   • Write everything into self.output.lock().unwrap(); end with \n (multi-line output convention from commit 2910884).
4. Special-token names (:, VARIABLE, ', …): after dictionary miss, check a small const list of known interpreter tokens (source: match arms at outer.rs:755-820, 927-992) and print SEE-IR: <name> is handled directly by the outer interpreter instead of erroring.

Documentation references: WORDS pattern outer.rs:6301-6310, 5328, 6045-6074; error style outer.rs:4057; output convention outer.rs:6056-6073.

Verification checklist:

• Tests (in outer.rs test module, eval_output style, cf. outer.rs:9035-9041):
  • : SQ DUP * ; SEE-IR SQ output contains dup and mul;
  • : FOO SQ SQ ; SEE-IR FOO shows the inlined body (contains two mul, no call) — locks in the "optimized view" semantics;
  • SEE-IR DUP works (IR primitive); SEE-IR WORDS prints host-word stub; SEE-IR NOSUCHWORD errors with SEE-IR: unknown word: NOSUCHWORD; bare SEE-IR errors with expected word name;
  • SEE-IR : prints the interpreter-token message.
• IF/ELSE/THEN and DO LOOP bodies render indented (one structured-word test).
• cargo test --workspace green; fmt + clippy clean; cargo check -p wafer-core --no-default-features passes.

Anti-pattern guards: do not print via a nonexistent HostAccess emit; do not gate name parsing on state == 0 (mandatory arg, not optional filter); do not THROW -13 (plain bail! matches TO/SYNONYM precedent).

---

Phase 3 — Source capture + SEE

Goal: SEE name prints the original source text : name … ; for colon words, synthesized definitions for data words, graceful stubs otherwise. This is the user-facing SEE.

What to implement:

1. Capture fields on ForthVM (near compiling_ir, outer.rs:204):

   compiling_source: String,      // accumulated raw text of the definition in progress
   source_capture_from: Option<usize>,  // input_pos where capture started in the CURRENT buffer
   word_sources: HashMap<WordId, String>,
   

2. Capture protocol (verbatim source, including comments and string literals — token-level reassembly would lose them):
   • start_colon_def() (outer.rs:2097): set source_capture_from to the position where : began. interpret_token() receives the token already consumed, so record the position before dispatch: in the evaluate() loop (outer.rs:518-521), remember pos_before = self.input_pos minus token — simplest correct form: capture token_start inside next_token() (outer.rs:690-704) into a new field last_token_start: usize as it skips whitespace; start_colon_def then does self.source_capture_from = Some(self.last_token_start).
   • End of evaluate() (after the loop, outer.rs:~536): if still compiling and capture active, flush input_buffer[from..] + '\n' into compiling_source, reset source_capture_from = Some(0) so the next buffer continues capture from its start.
   • finish_colon_def() (outer.rs:2266): flush input_buffer[from..=pos of ';'], store word_sources.insert(word_id, normalized), clear capture state. Normalize only trailing whitespace; keep interior verbatim.
   • Error path outer.rs:523-535: clear both capture fields alongside the existing compile-state wipe.
   • :NONAME and quotations (outer.rs:759-761, 773-778): skip capture (no name to SEE) — guard on compiling_name.is_some().
3. MARKER integration: add word_sources to MarkerState (outer.rs:166-176), snapshot (outer.rs:3462-3480) and restore (outer.rs:3484-3506) exactly as ir_bodies is handled there.
4. SEE word: pending code 41, same registration/dispatch shape as Phase 2. do_see() resolution order:
   1. word_sources hit → print stored source verbatim, append immediate on its own line if flagged (cf. set_immediate, dictionary.rs:404).
   2. Recognizable data-word IR shape (Phase 0 kind table) → synthesized one-liner. CONSTANT [PushI32(v)] → <v> CONSTANT <NAME>; VARIABLE → VARIABLE <NAME> ( addr=<v> ); VALUE [PushI32(a), Fetch] → <cur> VALUE <NAME> reading current value via self.rt memory read if cheap, else VALUE <NAME>; SYNONYM [Call(id)] → SYNONYM <NAME> <OLD>; DEFER → DEFER <NAME> plus current target name via does/pfa lookup when resolvable.
   3. ir_bodies hit (primitive or pre-capture colon word) → \ <NAME> is a primitive; IR: + format_ir_with output (reuse Phase 1/2 machinery — SEE never dead-ends).
   4. host_word_names hit → <NAME> is a built-in host word.
   5. Interpreter-token list → <NAME> is handled by the outer interpreter (compiler word).
   6. Else → bail!("SEE: unknown word: {name}").
5. Boot words get sources for free: boot.fth definitions flow through the same evaluate()/finish_colon_def path, so SEE NIP etc. shows real boot source. Verify, don't assume — one test below.

Documentation references: compile-state lifecycle outer.rs:512-535, 2097-2121, 2266-2329; multi-line REPL driver main.rs:411-416; MarkerState outer.rs:166-176, 3462-3506.

Verification checklist:

• : SQ DUP * ; SEE SQ prints : SQ DUP * ; (verbatim, one line).
• Multi-line: inline-VM test (pattern outer.rs:9077-9080): evaluate(": TRI\") then evaluate(\" DUP DUP ;"), then SEE TRI shows both lines.
• Comment survives: : C ( n -- n ) 1+ ; SEE C output contains ( n -- n ).
• 42 CONSTANT A SEE A → 42 CONSTANT A; VARIABLE V SEE V → contains VARIABLE V.
• SEE NIP (boot word) prints a colon definition, not an IR dump.
• SEE DUP prints the primitive-IR fallback; SEE WORDS prints host stub; SEE ' prints interpreter-token message; unknown word errors in house style.
• MARKER round-trip: define word, set marker, redefine, execute marker, SEE shows the original — plus existing marker tests still green.
• Immediate flag: : I2 ; IMMEDIATE SEE I2 output contains immediate.
• Error path: force unknown word mid-definition, then define a fresh word — its captured source must not contain debris from the aborted definition.
• Full suite + fmt + clippy + --no-default-features check.

Anti-pattern guards: do not reconstruct source from tokens (loses comments/strings/spacing); do not capture into word_sources for :NONAME; do not forget the error-path wipe (outer.rs:523-535) — stale capture corrupts the next definition's source; evaluate() resets input_pos per call (outer.rs:512-514) so source_capture_from is per-buffer, never carried across calls uncleared.

---

Phase 4 — HELP word + doc table

Goal: HELP name prints stack effect + one-line description; HELP alone prints usage. Shares lookup/classification with SEE.

What to implement:

1. New feature-free module crates/core/src/wordhelp.rs: a static table

   /// (NAME, stack effect, one-line description)
   pub const WORD_DOCS: &[(&str, &str, &str)] = &[
       ("DUP", "( x -- x x )", "Duplicate the top of the data stack."),
       ...
   ];
   pub fn lookup(name: &str) -> Option<(&'static str, &'static str)>  // case-insensitive
   

   Seed from the Forth 2012 glossary (stack effects are standardized). Cover, in priority order: core + core-ext words WAFER implements, then tools/double/float sets. Incomplete coverage is acceptable and expected — HELP says no help for <name> (word exists) when the word is defined but undocumented, which doubles as the TODO list.
2. HELP word: pending code 43, same registration/dispatch shape as Phase 2. Resolution: parse optional name (bare HELP → usage line HELP <word> — also try: WORDS, SEE <word>, SEE-IR <word>); table hit → print NAME ( stack effect ) description; miss but dictionary hit → no help for <name> + hint try SEE <name>; miss both → house-style unknown-word error.
3. Cross-wiring (the "as useful as possible" part):
   • SEE/SEE-IR prepend the HELP line as a \ ... comment when the table has one.
   • HELP appends immediate / built-in / defined in boot.fth or user code classification reusing the Phase 2/3 classifier — factor that classifier into a shared fn classify_word(&self, name) -> WordClass when Phase 4 lands (do NOT pre-build it in Phase 2; extract once there are two users, per smallest-change rule).
4. User-defined words: optional docstring convention — if the captured source's first parenthesized comment looks like a stack effect (( ... -- ... )), HELP echoes it for user words. No new syntax, zero cost, rewards idiomatic Forth style.

Verification checklist:

• HELP DUP prints stack effect + description; HELP dup (lowercase) same.
• HELP alone prints usage; HELP NOSUCH errors house-style; HELP MYWORD for undocumented-but-defined word prints the no help + SEE hint.
• : SQ ( n -- n^2 ) DUP * ; HELP SQ echoes ( n -- n^2 ).
• Table lint test: iterate WORD_DOCS, assert every documented name resolves in a booted VM's dictionary (catches typos/renames mechanically).
• Full suite + fmt + clippy + --no-default-features.

Anti-pattern guards: no doc strings threaded through register_primitive call sites (200+ call-site churn, bloats outer.rs — the side table is deliberate); no partial-coverage panic — missing docs degrade gracefully.

---

Phase 5 — Final verification + docs

1. Full gate: cargo fmt --all && cargo clippy --workspace (zero warnings) && cargo test --workspace (expect baseline 431 unit + new SEE/SEE-IR/HELP tests, 1 benchmark, 11 compliance, 9 comparison — all green).
2. Feature-freedom proof: cargo check -p wafer-core --no-default-features and web build cd crates/web && wasm-pack build --target web --dev --out-dir www/pkg.
3. Manual REPL pass (CLI): SEE SQ, SEE-IR FOO with inlining, HELP DUP, multi-line definition then SEE, tab-complete SE<tab> — confirm multi-line output renders per commit 2910884 conventions (block output, ok on own line).
4. Web REPL smoke: serve crates/web/www, run the same commands — output flows through take_output() (web/src/lib.rs:38-43), no web-side changes expected.
5. Anti-pattern grep: grep -n "cfg(feature" crates/core/src/see.rs crates/core/src/wordhelp.rs → empty; grep -n "impl Display for IrOp" -r crates/core → empty; grep -rn "emit" crates/core/src/see.rs → empty.
6. Docs: docs/FORTH.md:95 already lists SEE under Programming-Tools — verify claim now true; add SEE/SEE-IR/HELP to README feature list if words are enumerated there; extend CLAUDE.md test-count line.
7. Compliance untouched: cargo test -p wafer-core --test compliance — must stay 11/11 (suite excludes SEE by design, toolstest.fth:38-39).

---

Deliberate scope cuts (revisit later, not now)

• SEE-WASM (disassemble compiled module via wasmprinter): compiled bytes are likely dropped after instantiation; codegen.rs unexamined. Separate plan if wanted.
• IR→Forth source reconstruction for optimized bodies: lossy and misleading post-inlining; the source-capture path makes it unnecessary.
• LOCATE / editor integration: needs file/line provenance in the dictionary; out of scope.
• Forth-side doc syntax (:doc): revisit after self-hosting work starts; the ( n -- n^2 ) echo in Phase 4 covers the 80% case with zero syntax.