Tokenizer inputs a Str and output a list of Tokens
Slots
-
private Str buf
-
private TokenVal ch()
Parse a char literal token.
-
private Int col := 1
-
private Void consume()
Consume the cur char and advance to next char in buffer:
- updates cur and peek fields
- updates the line and col count
- end of file, sets fields to 0
-
private Int cur
-
private TokenVal? docComment()
Parse a Javadoc style comment into a documentation comment token.
-
private TokenVal dsl()
Parse a domain specific language <| ... |>
-
private Bool endOfQuoted(Quoted q)
If at end of quoted literal consume the ending token(s) and return true.
- errSource
-
override CompilerErr err(Str msg, Location? loc := null)
Overrides compiler::CompilerSupport.err
Return a CompilerException for current location in source.
- escapeSource
-
Int escape()
Parse an escapse sequence which starts with a \
-
private Str filename
-
private TokenVal? find()
Find the next token or return null.
- hexSource
-
TokenVal hex()
Process hex int/long literal starting with 0x
-
private Bool inStrLiteral
-
private Bool interpolation(Str s, Quoted q)
When we hit a $ inside a string it indicates an embedded expression. We make this look like a stream of tokens such that:
"a ${b} c" -> "a " + b + " c"
Return true if more in the string literal.
-
private Bool isDoc
-
private Int lastLine
-
private Int line := 1
- makeSource
-
new make(Compiler compiler, Location location, Str buf, Bool isDoc)
Construct with characters of source file. The buffer passed must be normalized in that all newlines must be represented strictly as \n and not \r or \r\n (see File.readAllStr). If isDoc is false, we skip all star-star Fandoc comments.
-
private TokenVal makeVirtualToken(Token kind, Obj? value := null)
Create a virtual token for string interpolation.
- nextSource
-
TokenVal? next()
Return the next token in the buffer.
-
private TokenVal number()
Parse a number literal token: int, float, decimal, or duration.
-
private Int peek
-
private Int peekPeek()
Peek at the character after peek
-
private Int pos
-
private Int posOfLine
-
private TokenVal? quoted(Quoted q)
Parse a quoted literal token: normal, triple, or uri Opening quote must already be consumed.
-
private TokenVal? skipCommentML()
Skip a multi line /* comment. Note unlike C/Java, slash/star comments can be nested.
-
private TokenVal? skipCommentSL()
Skip a single line // comment
-
private Bool skipStrWs(Int openLine, Int openPos)
Leading white space in a multi-line string is assumed to be outside of the string literal. If there is an non-whitespace char, then it is an compile time error. Return true if ok, false on error.
-
private TokenVal symbol()
Parse a symbol token (typically into an operator).
- tokenizeSource
-
TokenVal[] tokenize()
Tokenize the entire input into a list of tokens.
-
private TokenVal[] tokens
-
private Bool whitespace
-
private TokenVal word()
Parse a word token: alpha (alpha|number)* Words are either keywords or identifiers