
Tokenizer inputs a Str and output a list of Tokens
Slots
-
private Str buf
-
private TokenVal ch()
Parse a char literal token.
-
private Int col := 1
-
private Void consume()
Consume the cur char and advance to next char in buffer:
- updates cur and peek fields
- updates the line and col count
- end of file, sets fields to 0
-
private Int cur
-
private TokenVal docComment()
Parse a Javadoc style comment into a documentation comment token.
- err
-
override CompilerErr err(Str msg, Location loc := null)
Return a CompilerException for current location in source.
- escape
-
Int escape()
Parse an escapse sequence which starts with a \
-
private Str filename
-
private TokenVal find()
Find the next token or return null.
- hex
-
TokenVal hex()
Process hex int/long literal starting with 0x
-
private Bool inStrLiteral
-
private Bool isDoc
-
private Int lastLine
-
private Int line := 1
- main
-
static Void main()
- make
-
new make(Compiler compiler, Location location, Str buf, Bool isDoc)
Construct with characters of source file. The buffer passed must be normalized in that all newlines must be represented strictly as \n and not \r or \r\n (see File.readAllStr). If isDoc is false, we skip all star-star Fandoc comments.
-
private TokenVal makeVirtualToken(Token kind, Obj value := null)
Create a virtual token for string interpolation.
-
private TokenVal next()
Return the next token in the buffer.
-
private TokenVal number()
Parse a number literal token: int, float, decimal, or duration.
-
private Int peek
-
private Int pos
-
private TokenVal rawStr()
Parse a raw string literal token.
-
private TokenVal skipCommentML()
Skip a multi line /* comment. Note unlike C/Java, slash/star comments can be nested.
-
private TokenVal skipCommentSL()
Skip a single line // comment
-
private TokenVal str()
Parse a string literal token.
-
private Bool strInterpolation(Str s)
When we hit a $ inside a string it indicates an embedded expression. We make this look like a stream of tokens such that:
"a ${b} c" -> "a " + b + " c"
Return true if more in the string literal.
-
private TokenVal symbol()
Parse a symbol token (typically into an operator).
- tokenize
-
TokenVal[] tokenize()
Tokenize the entire input into a list of tokens.
-
private TokenVal[] tokens
-
private TokenVal uri()
Parse a uri literal token.
-
private TokenVal word()
Parse a word token: alpha (alpha|number)* Words are either keywords or identifiers