Fantom

 

class

compiler::Tokenize

sys::Obj
  compiler::CompilerSupport
    compiler::CompilerStep
      compiler::Tokenize

Tokenize is responsible for parsing all the source files into a a list of tokens. Each source file is mapped to a CompilationUnit and stored in the PodDef.units field:

Compiler.srcFiles -> Compiler.pod.units

During the standard pipeline this step is called by the InitInput step.

Slots

inputSource

CompilerInput input

makeSource

new make(Compiler compiler)

runSource

override Void run()

Run the step

tokenizeSource

CompilationUnit tokenize(Loc loc, Str src)