class
compiler::Tokenize
sys::Obj compiler::CompilerSupport compiler::CompilerStep compiler::Tokenize
Tokenize is responsible for parsing all the source files into a a list of tokens. Each source file is mapped to a CompilationUnit and stored in the PodDef.units field:
Compiler.srcFiles -> Compiler.pod.units
During the standard pipeline this step is called by the InitInput step.
Slots
- inputSource
-
CompilerInput input
- makeSource
-
new make(Compiler compiler)
- runSource
-
override Void run()
Overrides compiler::CompilerStep.run
Doc inherited from compiler::CompilerStep.run
Run the step
-
private Void runFileMode()
-
private Void runStrMode()
- tokenizeSource
-
CompilationUnit tokenize(Loc loc, Str src)