lexer - Factor Documentation

Vocabulary
lexer

Variable description
Stores the current lexer instance.

Class description
An object for tokenizing parser input. It has the following slots:

text the lines being parsed; an array of strings

line the line number being parsed; unlike most indices this is 1-based for friendlier error reporting and integration with text editors

column the current column position, zero-based

Custom lexing can be implemented by delegating a tuple to an instance of this class and implementing the skip-word and skip-blank generic words.

Definition

USING: arrays math strings vectors ;

IN: lexer

TUPLE: lexer
{ text array initial: { } } { line fixnum initial: 0 }
{ line-text string initial: "" }
{ line-length fixnum initial: 0 }
{ column fixnum initial: 0 }
{ parsing-words vector initial: V{ } } ;

Methods

USING: kernel lexer lexer.private sequences ;

M: lexer skip-blank
shebang?
[ [ nip length ] change-lexer-column ]
[ [ (skip-blank) ] change-lexer-column ] if ;

USING: lexer lexer.private ;

M: lexer skip-word [ (skip-word) ] change-lexer-column ;