Fan

 

class

compiler::Tokenizer

sys::Obj
  compiler::CompilerSupport
    compiler::Tokenizer

Tokenizer inputs a Str and output a list of Tokens

Slots

errSource

override CompilerErr err(Str msg, Location? loc := null)

Return a CompilerException for current location in source.

escapeSource

Int escape()

Parse an escapse sequence which starts with a \

hexSource

TokenVal hex()

Process hex int/long literal starting with 0x

makeSource

new make(Compiler compiler, Location location, Str buf, Bool isDoc)

Construct with characters of source file. The buffer passed must be normalized in that all newlines must be represented strictly as \n and not \r or \r\n (see File.readAllStr). If isDoc is false, we skip all star-star Fandoc comments.

nextSource

TokenVal? next()

Return the next token in the buffer.

tokenizeSource

TokenVal[] tokenize()

Tokenize the entire input into a list of tokens.