Source Edit

This module implements a base object of a lexer with efficient buffer handling. Only at line endings checks are necessary if the buffer needs refilling.

Imports

strutils, streams, assertions

Types

BaseLexer = object of RootObj bufpos*: int ## the current position within the buffer buf*: string ## the buffer itself ## the input stream lineNumber*: int ## the current line number offsetBase*: int: the base lexer. Inherit your lexer from this object. Source Edit

Consts

EndOfFile = '\x00': end of file marker Source Edit
NewLines = {'\r', '\n'}: Source Edit

Procs

proc close(L: var BaseLexer) {....raises: [IOError, OSError], tags: [WriteIOEffect], forbids: [].}: closes the base lexer. This closes L's associated stream too. Source Edit
proc getColNumber(L: BaseLexer; pos: int): int {....raises: [], tags: [], forbids: [].}: retrieves the current column. Source Edit
proc getCurrentLine(L: BaseLexer; marker: bool = true): string {....raises: [], tags: [], forbids: [].}: retrieves the current line. Source Edit
proc handleCR(L: var BaseLexer; pos: int): int {....raises: [IOError, OSError], tags: [ReadIOEffect], forbids: [].}: Call this if you scanned over '\c' in the buffer; it returns the position to continue the scanning from. pos must be the position of the '\c'. Source Edit
proc handleLF(L: var BaseLexer; pos: int): int {....raises: [IOError, OSError], tags: [ReadIOEffect], forbids: [].}: Call this if you scanned over '\L' in the buffer; it returns the position to continue the scanning from. pos must be the position of the '\L'. Source Edit
proc handleRefillChar(L: var BaseLexer; pos: int): int {. ...raises: [IOError, OSError], tags: [ReadIOEffect], forbids: [].}: Call this if a terminator character other than a new line is scanned at pos; it returns the position to continue the scanning from. Source Edit
proc open(L: var BaseLexer; input: Stream; bufLen: int = 8192; refillChars: set[char] = NewLines) {....raises: [IOError, OSError], tags: [ReadIOEffect], forbids: [].}: inits the BaseLexer with a stream to read from. Source Edit

std/lexbase

Imports

Types

Consts

Procs