Asama 0.3 — Add SourceLocation to all Token types #37

New Issue

saqut · 2026-05-26T19:51:33+03:00

saqut commented

2026-05-26 19:51:33 +03:00

Goal: Every token produced by the Tokenizer carries its SourceLocation.

Files to modify:

src/tokenizer/token.hpp

Requirements:

Add SourceLocation loc field to the base Token class.
Remove or deprecate int start, int end fields (replace with loc data).
Update StringToken, NumberToken, IdentifierToken if they reference start/end directly.
Ensure all token constructors initialize loc.

Success criteria:

After tokenizing, every token has a valid SourceLocation.
Old start/end offsets are derivable from SourceLocation if needed.

**Goal:** Every token produced by the Tokenizer carries its SourceLocation. **Files to modify:** - src/tokenizer/token.hpp **Requirements:** - Add SourceLocation loc field to the base Token class. - Remove or deprecate int start, int end fields (replace with loc data). - Update StringToken, NumberToken, IdentifierToken if they reference start/end directly. - Ensure all token constructors initialize loc. **Success criteria:** - After tokenizing, every token has a valid SourceLocation. - Old start/end offsets are derivable from SourceLocation if needed.

saqut closed this issue

2026-06-14 21:53:28 +03:00

Sign in to join this conversation.