Table of Contents

Module: FileParsing ./src/peak/util/

Line-Oriented File Parsing Tools

This module supplies functions for creating and using "lineInfo" streams, which are iterables of (source,lineNo,line) tuples. The source is an indicator of the line's origin (e.g. a filename), while lineNo is its line number within that source. line is the actual text of the line. This data structure is simple, fast, and easy to use.

This module was created because the standard library ConfigParser makes a lot of assumptions about syntax that don't necessarily work with every config file. For example, it assumes directives are order-insensitive key-value pairs, which doesn't work well for systems like PEAK, which might prefer to process a stream of directives in the original sequence.

So, this module uses a stream-processing approach that should provide a reusable foundation for other types of line-oriented text processing. The basic tools are the fromStream(), fromFile(), and fromString() functions, which respectively take an iterable, a filename, or a string, and return an iterator that yields (source,lineNo,line) tuples.

Once you have a stream to play with, you can then use some of the processors such as iterConfigSections() and iterConfigSettings(), which return iterators yielding various kinds of configuration data (based on ConfigParser-like syntax rules). Or, you can use the AbstractConfigParser class as a base class to create your own specialized parsers.

Imported modules   
from __future__ import generators
import re
fromFile ( filename,  mode='r' )

Produce (source,lineNo,line) tuple stream from input file

This is the equivalent of fromStream(open(filename,mode), filename). That is, it returns a line-info iterator with a source of filename and the lines from open(filename,mode).readlines().

fromStream ( stream,  source=None )

Produce (source,lineNo,line) tuple stream from input lines

Calling fromStream(stream,source) returns an iterator which yields (source,lineNo,line) tuples for each line in stream.

stream must be a sequence, iterator, or iterable file-like object that yields text lines. (Line ending characters are stripped from line ends.)

source should be a short string (e.g. filename) or other useful identifier of where the lines came from.

fromString ( text,  source='<string>' )

Produce (source,lineNo,line) tuple stream from input file

This is the equivalent of fromStream(StringIO(text), source). That is, it returns a line-info iterator with the supplied source name (default is "<string>") and the lines from StringIO(text).readlines().

iterConfigSections ( lineSource )

(section,lines,info) tuples per .ini-like section in lineSource

This function is used to break up a configuration file (.ini or ConfigParser-style) into sections based on '[]-enclosed section names. It returns an iterator which yields (section,lines,info)' tuples for each section in the file. The first yielded section will be None if any lines appear before the first section heading; all others will be the string that was between the [].

lines is always a list of (source,lineNo,line) tuples, suitable for use by iterConfigSettings() or other lineInfo stream processors.

info is a (source,lineNo,line) tuple representing the line where the section header (if any) appeared.

iterConfigSettings ( lineSource )

(name,value,lineInfo) tuples per .ini-like setting in lineSource

name and value will be None for any non-blank, non-comment line which does not appear to be a valid option. Otherwise, they are the setting's name and value, respectively.

lineInfo is a standard lineInfo-tuple of (source,lineNo,line) data, with the difference that continuation lines are concatenated to line. This is so that if one needs to display an error message that shows the source of the parsed value, the full logical line is included, even though the first physical line number would be used to identify the error line.

Continuation Lines

RFC822-style line continuations are supported, with leading whitespace stripped from continuation lines, and "\n" separating the lines in the returned value. Unlike ConfigParser, no other interpretation of name or value is done, so it's up to you to do any case-folding, conversions, etc.

Comment and Whitespace (blank line) Processing

Comment lines are lines which begin with a ';', #, or the word rem (case-insensitive). No leading whitespace is allowed, to prevent confusion with continuation lines. Because setting values are not interpreted, comments embedded on the same line with a setting or indented in a continuation line are returned as part of the value text. If you want to support embedded comments, it is up to you to parse them out of the value.

Comment lines are completely ignored, so you can have a comment line inside a series of continuation lines, as long as it has no leading whitespace on the line. Blank (empty or whitespace-only) lines within a series of continuation lines are considered part of the setting value, and are rendered as empty lines in the value. Blank lines which appear at the end of a setting value, or before the first setting in the input stream, are ignored.



Abstract configuration file parser based on sections and settings

Table of Contents

This document was automatically generated on Mon May 20 01:11:19 2019 by HappyDoc version 2.1