Index

Package: Scanners

Description

package Tokens.Scanners is
Copyright (c) 2012 Kevin Wellwood All rights reserved. This source code is distributed under the Modified BSD License. For terms and conditions, see license.txt.

Classes

Token_Scanner

type Token_Scanner is new Limited_Object with private;

Ancestors:

Primitive operations:

Delete (overriding Objects.Delete)
Objects.Construct (Inherited)
Objects.To_String (Inherited)
A Token_Scanner parses and scans a text stream for specific types of tokens. Tokens can be optionally accepted from the stream, or expected. If a specific token type is expected but a different type is found next, a Parse_Exception is raised and the token remains.

Types

A_Token_Scanner

type A_Token_Scanner is access all Token_Scanner'Class;

Constants & Global variables

Parse_Exception

Parse_Exception : exception;
Raised when an unexpected token is found.

Subprograms & Entries

Create_Token_Scanner

function Create_Token_Scanner return A_Token_Scanner;
Creates a new Token_Scanner.

Accept_Binary_Op

function Accept_Binary_Op
( this: not null access Token_Scanner'Class ) return A_Token;
Accepts a valid binary expression operator, as defined by the class Binary_Operator. If a valid binary operator is not found, no tokens are taken from the stream and null is returned. Token_Exception will be raised if an unrecognized or malformed token is encountered; The bad token will be discarded and not returned again.

Accept_Identifier

function Accept_Identifier
( this: not null access Token_Scanner'Class ) return A_Identifier_Token;
Optionally accepts an identifier token. Returns the token if the next in the stream is an identifier, otherwise null is returned. This is the same as calling Accept_Token( TK_IDENTIFIER ). Token_Exception will be raised if an unrecognized or malformed token is encountered; The bad token will be discarded and not returned again.

Accept_Token

function Accept_Token
( this: not null access Token_Scanner'Class;
tokenType: Token_Type ) return A_Token;
Optionally accepts a token of a specific type. Returns the next token from the stream if it matches 'tokenType', otherwise the token remains in the stream and null is returned. Token_Exception will be raised if an unrecognized or malformed token is encountered; The bad token will be discarded and not returned again.

Accept_Token

function Accept_Token
( this: not null access Token_Scanner'Class;
tokenType: Token_Type ) return Boolean;
Same as the functional Accept_Token, but True is returned if 'tokenType' matches the next token in the stream and False is returned if it does not. If the token is accepted, it will be removed from the token stream and discarded instead of being returned. Token_Exception will be raised if an unrecognized or malformed token is encountered; The bad token will be discarded and not returned again.

Accept_Unary_Op

function Accept_Unary_Op
( this: not null access Token_Scanner'Class ) return A_Token;
Optionally accepts a valid unary expression operator, as defined by the class Unary_Operator. If a valid unary operator is not found, no tokens are taken from the stream and null is returned. Token_Exception will be raised if an unrecognized or malformed token is encountered; The bad token will be discarded and not returned again.

Expect_Identifier

function Expect_Identifier
( this: not null access Token_Scanner'Class ) return A_Identifier_Token;
Requires an identifier token to be next. If the next token in the stream is not an identifier then Parse_Exception will be raised. Token_Exception will be raised if an unrecognized or malformed token is encountered; The bad token will be discarded and not returned again.

Expect_Token

function Expect_Token
( this: not null access Token_Scanner'Class;
tokenType: Token_Type ) return A_Token;
Requires the next token to be of a specific type. Returns the next token from the stream if it matches 'tokenType', otherwise the token remains in the stream and Parse_Exception is raised. Token_Exception will be raised if an unrecognized or malformed token is encountered; The bad token will be discarded and not returned again.

Expect

procedure Expect
( this: not null access Token_Scanner'Class;
tokenType: Token_Type );
Requires the next token to be of a specific type. If the next token does not match 'tokenType', it will remain in the stream and a Parse_Exception will be raised. Otherwise, the expected token will be consumed and the procedure will return normally. Token_Exception will be raised if an unrecognized or malformed token is encountered; The bad token will be discarded and not returned again.

Get_Location

function Get_Location
( this: not null access Token_Scanner'Class ) return Token_Location;
Returns the current location of the scanner in the input stream.

Push_Back

procedure Push_Back
( this: not null access Token_Scanner'Class;
token: in out A_Token );
Returns a token to the front of the token stream. 'token' is consumed.

Set_Input

procedure Set_Input
( this: not null access Token_Scanner'Class;
stream: Stream_Access );
Sets the input stream for reading characters. If 'stream' is null, the scanner's input will be cleared.

Delete

procedure Delete
( this: in out A_Token_Scanner );
Deletes the Token_Scanner;