Index

Package: Tokenizers

Description

package Tokens.Tokenizers is

Classes

Tokenizer

type Tokenizer is new Limited_Object with private;

Ancestors:

Primitive operations:

Delete (overriding Objects.Delete)
Objects.Construct (Inherited)
Objects.To_String (Inherited)
A Tokenizer reads from an input stream and splits the text using a set of token-delimiter characters to returns a series of Token objects. If the stream contains character sequences that are not recognizable tokens, or are malformed tokens, an exception will be raised and the erroneous text discarded.

Types

A_Tokenizer

type A_Tokenizer is access all Tokenizer'Class;

Constants & Global variables

Token_Exception

Token_Exception : exception;
Raised when a token is malformed or unrecognized.

Subprograms & Entries

Create_Tokenizer

function Create_Tokenizer return A_Tokenizer;
Creates a new Tokenizer.

Get_Location

function Get_Location
( this: not null access Tokenizer'Class ) return Token_Location;
Returns the tokenizer's current location in the input stream (or the location of the next token to be returned, if tokens have been pushed back.)

Get_Next

function Get_Next
( this: not null access Tokenizer'Class ) return A_Token;
Returns the next Token read from the input stream. If tokens were previously pushed back to the tokenizer, the most recent will be returned now instead. Token_Exception will be raised if the input stream contents can't be recognized as a token or the token is malformed.

Push_Back

procedure Push_Back
( this: not null access Tokenizer'Class;
token: in out A_Token );
Returns the most recent Token received from the Tokenizer. The next call to Get_Next will return 'token' instead of reading from the token stream. Multiple tokens can be returned to the Tokenizer, in the order in which they were originally received by the caller. 'token' will be consumed.

Set_Input

procedure Set_Input
( this: not null access Tokenizer'Class;
stream: Stream_Access );
Sets the input stream for reading characters. The state of the Tokenizer will also be reset. If 'stream' is null, the Tokenizer's input will be cleared.

Delete

procedure Delete
( this: in out A_Tokenizer );
Deletes the Tokenizer.