logo
class

compiler::Tokenizer

sys::Obj
  compiler::CompilerSupport
    compiler::Tokenizer

Tokenizer inputs a Str and output a list of Tokens

Slots

err

virtual CompilerErr err(Str msg, Location loc := def)

Return a CompilerException for current location in source.

escape

Int escape()

Parse an escapse sequence which starts with a \

hex

TokenVal hex()

Process hex int/long literal starting with 0x

main

static const Void main()

make

new make(Compiler compiler, Location location, Str buf, Bool isDoc)

Construct with characters of source file. The buffer passed must be normalized in that all newlines must be represented strictly as \n and not \r or \r\n (see File.readAllStr). If isDoc is false, we skip all star-star Fandoc comments.

tokenize

TokenVal[] tokenize()

Tokenize the entire input into a list of tokens.