help
XMLTreeWalker Documentation
XMLTreeWalker
Purpose
This document describes how to use the XmlTreeWalker product. XmlTreeWalker works in conjunction with expat. It can be used for filtering and content modification of XML documents by employing various Operators and having them activate based on specified Filter rules. Uses include content extraction, suppression and rewriting.
This is an initial release. It is also my first distribution ever, so it's likely to have various issues.
The following example takes an xhtml document and parses the table of contents. The PrintOperator and the DocbookTocOperator both filter the same part of the tree. This part is outlined below.
<html>
<body>
<div class="article">
<div class="toc">
<dl>
parsed content including the dl tag
</dl>
</div>
</div>
</body>
</html>
The PrintOperator reprints everything except the parsed content. The DocbookTocOperator only deals with the parsed content and extracts out a
<dl>
<dt>
<a href="url">
name
</a>
</dt>
<dd>
<dl>
<dt>
....
type definition list and returns a recursive list of TocItems representing the TOC data parsed.
The following is the code that accomplishes that.
import codecs
# xmltreewalker uses unicode so we need to decode the utf-8
in = codecs.open(path, 'rb', 'utf-8')
orig = in.read()
in.close()
# setup the filter for toc processing
flt = Filter.getInstance([['html'], ['body'],
['div', {'class': 'article'}],
['div', {'class':'toc'}],
['dl'],
])
dto = DocbookTocOperator(False, [flt])
# this filter allows everything but the TOC
flt2 = Filter.getInstance([['html'], ['body'],
['div', {'class': 'article'}],
['div', {'class':'toc'}],
])
# the output of the PrintOperator will be written to a file like object
# using data.write(content). For larger datasets we could just as well use the output
# file opened below and write right to that.
data = StringIO()
# set up the PrintOperator to print unless the filter is met
# the filter list is subsequently specified along with where the
# data goes to.
pto = PrintOperator(True, [flt2], data)])
# this sets up the treewalker with it's operators
wlk = XMLTreeWalker([dto, pto])
# this makes things happen
wlk.parse(orig)
# retrieve the parsed data from the PrintOperator
got = data.getvalue()
data.close()
# write it back remembering that it is unicode and needs to
# be re-encoded into utf-8
output = codecs.open(outPath, 'w', 'utf-8')
output.write(got)
output.close()
# This prints a pretty version of the TOC
print >>> sys.stdout(dto.rootItem.prettyPrint())
A pretty printed version of that could look something like:
URL: None - Name: Root - Children:
URL: #d0e24 - Name: Purpose
URL: #d0e29 - Name: Status
URL: #d0e34 - Name: Requirements
URL: #d0e44 - Name: Usage
XMLTreeWalker registers its handlers with expat and dispatches the calls to all active operators in the order in which they were specified in the list. Each operator returns a Boolean indicating whether to continue to process the other operators (True) or to abort calling the other operators (False). This is useful for masking. One can place an operator to remove data first in the list and activate it as needed. This way when it's active it can terminate processing for all subsequent operators, effectively masking sections of a document out.
Operators act on the data they get when they're active. Whether they are or not depends on their filters and the initial state. The initial state is specified in the constructor. If isActive is True the operator starts out active and processes data until one of the filters is met. At that time the state is toggled to inactive until the filter condition no longer applies. Starting an operator with isActive set to False accomplishes the exact opposite.
Multiple filters can be specified for an operator to activate or deactivate. They are specified as a list and are processed in that order. When any of the filters match the isActive state of the operator is toggled.
Filter specification always starts at the root node of the document. They may be specified as a series of TagRequirements which include the tag name and a dictionary of property value pairs. The tag name may be set to None in which case any tag will match. The properties may be specified along with values or with their value set to None for any. For a TagRequirement to be met, all specified properties must match unless they're set to None.
