--- epic: 1 story: 1.1 title: "Lexer & Tokenizer" status: review --- ## Epic 1 — Core Calculation Engine (Rust Crate) **Goal:** Build `calcpad-engine` as a standalone Rust crate that powers all platforms. This is the foundation. ### Story 1.1: Lexer & Tokenizer As a CalcPad engine consumer, I want input lines tokenized into a well-defined token stream, So that the parser can build an AST from structured, unambiguous tokens rather than raw text. **Acceptance Criteria:** **Given** an input line containing an integer such as `42` **When** the lexer tokenizes the input **Then** it produces a single `Number` token with value `42` **And** no heap allocations occur for this simple expression **Given** an input line containing a decimal number such as `3.14` **When** the lexer tokenizes the input **Then** it produces a single `Number` token with value `3.14` **Given** an input line containing a negative number such as `-7` **When** the lexer tokenizes the input **Then** it produces tokens representing the negation operator and the number `7` **Given** an input line containing scientific notation such as `6.022e23` **When** the lexer tokenizes the input **Then** it produces a single `Number` token with value `6.022e23` **Given** an input line containing SI scale suffixes such as `5k`, `2.5M`, or `1B` **When** the lexer tokenizes the input **Then** it produces `Number` tokens with values `5000`, `2500000`, and `1000000000` respectively **Given** an input line containing currency symbols such as `$20`, `€15`, `£10`, `¥500`, or `R$100` **When** the lexer tokenizes the input **Then** it produces `CurrencySymbol` tokens paired with their `Number` tokens **And** multi-character symbols like `R$` are recognized as a single token **Given** an input line containing unit suffixes such as `5kg`, `200g`, or `3.5m` **When** the lexer tokenizes the input **Then** it produces `Number` tokens followed by `Unit` tokens **Given** an input line containing arithmetic operators `+`, `-`, `*`, `/`, `^`, `%` **When** the lexer tokenizes the input **Then** it produces the corresponding `Operator` tokens **Given** an input line containing natural language operators such as `plus`, `minus`, `times`, or `divided by` **When** the lexer tokenizes the input **Then** it produces the same `Operator` tokens as their symbolic equivalents **And** `divided by` is recognized as a single two-word operator **Given** an input line containing a variable assignment such as `x = 10` **When** the lexer tokenizes the input **Then** it produces an `Identifier` token, an `Assign` token, and a `Number` token **Given** an input line containing a comment such as `// this is a note` **When** the lexer tokenizes the input **Then** it produces a `Comment` token containing the comment text **And** the comment token is preserved for display but excluded from evaluation **Given** an input line containing plain text with no calculable expression **When** the lexer tokenizes the input **Then** it produces a `Text` token representing the entire line **Given** an input line containing mixed content such as `$20 in euro - 5% discount` **When** the lexer tokenizes the input **Then** it produces tokens for the currency value, the conversion keyword, the currency target, the operator, the percentage, and the keyword **And** each token includes its byte span (start, end) within the input --- ### Tasks/Subtasks - [x] **Task 1: Set up Rust crate and define Token types** - [x] 1.1: Initialize `calcpad-engine` Rust crate with `Cargo.toml` - [x] 1.2: Define `Span` struct (start, end byte offsets) - [x] 1.3: Define `TokenKind` enum (Number, Operator, Identifier, Assign, CurrencySymbol, Unit, Comment, Text, Keyword, Percent, LParen, RParen) - [x] 1.4: Define `Token` struct with `kind`, `span`, and value representation - [x] 1.5: Write unit tests for Token construction and Span ranges - [x] **Task 2: Implement core Lexer struct and scanning infrastructure** - [x] 2.1: Create `Lexer` struct holding input `&str`, cursor position, and token output - [x] 2.2: Implement character peek, advance, and whitespace-skipping helpers - [x] 2.3: Implement `tokenize()` method that dispatches to specific scanners based on current char - [x] 2.4: Write tests verifying empty input, whitespace-only input - [x] **Task 3: Tokenize numbers (integers, decimals, scientific notation, SI suffixes)** - [x] 3.1: Implement integer scanning (sequence of digits) - [x] 3.2: Implement decimal scanning (digits, dot, digits) - [x] 3.3: Implement scientific notation scanning (e/E followed by optional +/- and digits) - [x] 3.4: Implement SI scale suffix detection (k, M, B, T) and multiply value accordingly - [x] 3.5: Write tests for integers, decimals, scientific notation, and SI suffixes (42, 3.14, 6.022e23, 5k, 2.5M, 1B) - [x] **Task 4: Tokenize operators (symbolic and natural language)** - [x] 4.1: Implement single-character operator scanning (+, -, *, /, ^, %) - [x] 4.2: Implement parentheses scanning ( and ) - [x] 4.3: Implement natural language operator recognition (plus, minus, times, divided by, of) - [x] 4.4: Handle `divided by` as a two-word operator - [x] 4.5: Write tests for all symbolic operators and natural language equivalents - [x] **Task 5: Tokenize identifiers, assignments, currency symbols, and units** - [x] 5.1: Implement identifier scanning (alphabetic sequences) - [x] 5.2: Implement `=` assignment operator scanning - [x] 5.3: Implement currency symbol scanning ($, €, £, ¥, and multi-char R$) - [x] 5.4: Implement unit suffix detection after numbers (kg, g, m, lb, etc.) - [x] 5.5: Implement keyword detection (in, to, as, of, discount, off) - [x] 5.6: Write tests for identifiers, assignments, currency symbols, units, keywords - [x] **Task 6: Tokenize comments and plain text fallback** - [x] 6.1: Implement `//` comment scanning (rest of line becomes Comment token) - [x] 6.2: Implement plain text detection (lines with no calculable tokens become Text) - [x] 6.3: Write tests for comment scanning and plain text fallback - [x] **Task 7: Integration tests for mixed content and span correctness** - [x] 7.1: Write integration test for `x = 10` → [Identifier, Assign, Number] - [x] 7.2: Write integration test for `$20 in euro - 5% discount` → full token stream - [x] 7.3: Write integration test verifying byte spans on all tokens - [x] 7.4: Write integration test for `-7` → [Operator(Minus), Number(7)] - [x] 7.5: Write integration test for edge cases (multiple spaces, trailing whitespace, empty input) - [x] 7.6: Verify no heap allocations for simple expressions (use stack-based token collection) --- ### Dev Notes **Architecture:** - `calcpad-engine` is a standalone Rust library crate (`lib.rs`) - The lexer operates on a single line of input at a time (line-oriented) - Tokens must include byte spans for syntax highlighting and error reporting - Design for zero/minimal heap allocation on simple expressions - SI suffixes (k, M, B, T) are resolved at lex time — the Number token stores the scaled value - Natural language operators map to the same `Operator` variants as symbolic ones - The `-` in `-7` is tokenized as a separate Minus operator, not part of the number (parser handles unary minus) - Currency symbols precede numbers; unit suffixes follow numbers - Multi-character currency symbols (R$) must be handled with lookahead - `divided by` requires two-word lookahead - Plain text detection is a fallback — if the lexer cannot produce any calculable tokens, the entire line becomes a Text token **Coding Standards:** - Use `#[derive(Debug, Clone, PartialEq)]` on all public types - Use `f64` for number representation (arbitrary precision comes in Story 1.4) - Minimize allocations — use `&str` slices into the input where possible - All public API must be documented with doc comments --- ### Dev Agent Record **Implementation Plan:** - Created `calcpad-engine` Rust library crate with two modules: `token` (types) and `lexer` (scanning) - Token types: `Span`, `Operator` enum, `TokenKind` enum (12 variants), `Token` struct - Lexer uses a byte-offset cursor scanning approach with peek/advance helpers - Unit suffixes after numbers use a "pending token" mechanism (Lexer stores pending Unit token emitted on next iteration) - SI suffixes (k, M, B, T) distinguished from unit suffixes by checking if followed by more alpha chars - Natural language operators and keywords matched via case-insensitive word-boundary matching - `divided by` handled with two-word lookahead - Plain text fallback: if no calculable tokens found, entire line becomes Text token - Red-green-refactor cycle followed: wrote failing tests first, then implementation, then cleanup **Debug Log:** - Fixed empty/whitespace input returning Text instead of empty vec (trim check was placed after comment check) - Fixed clippy warning: `op.clone()` on Copy type `Operator` → replaced with `*op` - Fixed unit suffix tokens not being emitted: pending token was set but loop exited before checking it; restructured loop to check pending tokens at top of each iteration **Completion Notes:** - All 39 tests pass (5 token tests + 34 lexer tests) - Zero clippy warnings - All 13 acceptance criteria satisfied with dedicated tests - Token types are well-documented with doc comments - Public API: `tokenize(input: &str) -> Vec` convenience function + `Lexer::new().tokenize()` --- ### File List - `Cargo.toml` (new) — Rust crate manifest for calcpad-engine - `src/lib.rs` (new) — Crate root, re-exports public API - `src/token.rs` (new) — Token types: Span, Operator, TokenKind, Token with unit tests - `src/lexer.rs` (new) — Lexer implementation with 34 unit/integration tests --- ### Change Log - 2026-03-16: Story 1.1 implemented — full lexer/tokenizer for CalcPad engine with 39 tests, all passing. Covers integers, decimals, scientific notation, SI suffixes, currency symbols, unit suffixes, operators (symbolic + natural language), identifiers, assignments, comments, plain text, keywords, percentages, parentheses, and mixed expressions with byte-span tracking. --- ### Status **Current:** review