Doug Hellmann

PyMOTW: Parsing XML Documents with ElementTree

2010-03-14T10:58:00.001-04:00

Parsing XML Documents with ElementTree

Parsed XML documents are represented in memory by ElementTree and
Element objects connected into a tree structure based on the way the
nodes in the XML document are nested.

Parsing an Entire Document

When you parse an entire document with parse(), an ElementTree
instance is returned. The tree knows about all of the data in the
input document, and the nodes of the tree can be searched or
manipulated in place. While this flexibility can make working with
the parsed document a little easier, it typically takes more memory
than an event-based parsing approach since the entire document must be
loaded at one time.

The memory footprint of small, simple documents such as this list of
podcasts represented as an OPML outline is not significant:

<?xml version="1.0" encoding="UTF-8"?>
<opml version="1.0">
<head>
	<title>My Podcasts</title>
	<dateCreated>Sun, 07 Mar 2010 15:53:26 GMT</dateCreated>
	<dateModified>Sun, 07 Mar 2010 15:53:26 GMT</dateModified>
</head>
<body>
  <outline text="Science and Tech">
    <outline text="APM: Future Tense" type="rss" 
             xmlUrl="http://www.publicradio.org/columns/futuretense/podcast.xml" 
             htmlUrl="http://www.publicradio.org/columns/futuretense/" />
	<outline text="Engines Of Our Ingenuity Podcast" type="rss" 
             xmlUrl="http://www.npr.org/rss/podcast.php?id=510030" 
             htmlUrl="http://www.uh.edu/engines/engines.htm" />
	<outline text="Science &#38; the City" type="rss" 
             xmlUrl="http://www.nyas.org/Podcasts/Atom.axd" 
             htmlUrl="http://www.nyas.org/WhatWeDo/SciencetheCity.aspx" />
  </outline>
  <outline text="Books and Fiction">
	<outline text="Podiobooker" type="rss" 
             xmlUrl="http://feeds.feedburner.com/podiobooks" 
             htmlUrl="http://www.podiobooks.com/blog" />
	<outline text="The Drabblecast" type="rss" 
             xmlUrl="http://web.me.com/normsherman/Site/Podcast/rss.xml" 
             htmlUrl="http://web.me.com/normsherman/Site/Podcast/Podcast.html" />
	<outline text="tor.com / category / tordotstories" type="rss" 
             xmlUrl="http://www.tor.com/rss/category/TorDotStories" 
             htmlUrl="http://www.tor.com/" />
  </outline>
  <outline text="Computers and Programming">
	<outline text="MacBreak Weekly" type="rss" 
             xmlUrl="http://leo.am/podcasts/mbw" 
             htmlUrl="http://twit.tv/mbw" />
	<outline text="FLOSS Weekly" type="rss" 
             xmlUrl="http://leo.am/podcasts/floss" 
             htmlUrl="http://twit.tv" />
	<outline text="Core Intuition" type="rss" 
             xmlUrl="http://www.coreint.org/podcast.xml" 
             htmlUrl="http://www.coreint.org/" />
  </outline>
  <outline text="Python">
    <outline text="PyCon Podcast" type="rss" 
             xmlUrl="http://advocacy.python.org/podcasts/pycon.rss" 
             htmlUrl="http://advocacy.python.org/podcasts/" />
	<outline text="A Little Bit of Python" type="rss" 
             xmlUrl="http://advocacy.python.org/podcasts/littlebit.rss" 
             htmlUrl="http://advocacy.python.org/podcasts/" />
	<outline text="Django Dose Everything Feed" type="rss" 
             xmlUrl="http://djangodose.com/everything/feed/" />
  </outline>
  <outline text="Miscelaneous">
	<outline text="dhellmann's CastSampler Feed" type="rss" 
             xmlUrl="http://www.castsampler.com/cast/feed/rss/dhellmann/" 
             htmlUrl="http://www.castsampler.com/users/dhellmann/" />
  </outline>
</body>
</opml>

To parse the file, pass an open file handle to parse(). It will
read the data, parse the XML, and return an ElementTree object.

from xml.etree import ElementTree

with open('podcasts.opml', 'rt') as f:
    tree = ElementTree.parse(f)

print tree

$ python ElementTree_parse_opml.py
<xml.etree.ElementTree.ElementTree instance at 0x82f58>

Traversing the Parsed Tree

Now that we have a parsed XML tree, we can iterate over it, visiting
all of the children in order and examining their attributes and
contents.

from xml.etree import ElementTree

with open('podcasts.opml', 'rt') as f:
    tree = ElementTree.parse(f)

for node in tree.getiterator():
    print node.tag, node.attrib

Here we print the entire tree, one tag at a time.

$ python ElementTree_dump_opml.py
opml {'version': '1.0'}
head {}
title {}
dateCreated {}
dateModified {}
body {}
outline {'text': 'Science and Tech'}
outline {'xmlUrl': 'http://www.publicradio.org/columns/futuretense/podcast.xml', 'text': 'APM: Future Tense', 'type': 'rss', 'htmlUrl': 'http://www.publicradio.org/columns/futuretense/'}
outline {'xmlUrl': 'http://www.npr.org/rss/podcast.php?id=510030', 'text': 'Engines Of Our Ingenuity Podcast', 'type': 'rss', 'htmlUrl': 'http://www.uh.edu/engines/engines.htm'}
outline {'xmlUrl': 'http://www.nyas.org/Podcasts/Atom.axd', 'text': 'Science & the City', 'type': 'rss', 'htmlUrl': 'http://www.nyas.org/WhatWeDo/SciencetheCity.aspx'}
outline {'text': 'Books and Fiction'}
outline {'xmlUrl': 'http://feeds.feedburner.com/podiobooks', 'text': 'Podiobooker', 'type': 'rss', 'htmlUrl': 'http://www.podiobooks.com/blog'}
outline {'xmlUrl': 'http://web.me.com/normsherman/Site/Podcast/rss.xml', 'text': 'The Drabblecast', 'type': 'rss', 'htmlUrl': 'http://web.me.com/normsherman/Site/Podcast/Podcast.html'}
outline {'xmlUrl': 'http://www.tor.com/rss/category/TorDotStories', 'text': 'tor.com / category / tordotstories', 'type': 'rss', 'htmlUrl': 'http://www.tor.com/'}
outline {'text': 'Computers and Programming'}
outline {'xmlUrl': 'http://leo.am/podcasts/mbw', 'text': 'MacBreak Weekly', 'type': 'rss', 'htmlUrl': 'http://twit.tv/mbw'}
outline {'xmlUrl': 'http://leo.am/podcasts/floss', 'text': 'FLOSS Weekly', 'type': 'rss', 'htmlUrl': 'http://twit.tv'}
outline {'xmlUrl': 'http://www.coreint.org/podcast.xml', 'text': 'Core Intuition', 'type': 'rss', 'htmlUrl': 'http://www.coreint.org/'}
outline {'text': 'Python'}
outline {'xmlUrl': 'http://advocacy.python.org/podcasts/pycon.rss', 'text': 'PyCon Podcast', 'type': 'rss', 'htmlUrl': 'http://advocacy.python.org/podcasts/'}
outline {'xmlUrl': 'http://advocacy.python.org/podcasts/littlebit.rss', 'text': 'A Little Bit of Python', 'type': 'rss', 'htmlUrl': 'http://advocacy.python.org/podcasts/'}
outline {'xmlUrl': 'http://djangodose.com/everything/feed/', 'text': 'Django Dose Everything Feed', 'type': 'rss'}
outline {'text': 'Miscelaneous'}
outline {'xmlUrl': 'http://www.castsampler.com/cast/feed/rss/dhellmann/', 'text': "dhellmann's CastSampler Feed", 'type': 'rss', 'htmlUrl': 'http://www.castsampler.com/users/dhellmann/'}

If we wanted to print only the groups of names and feed URLs for the
podcasts, leaving out of all of the data in the header section, we
could iterate over only just the outline nodes and print the
text and xmlUrl attributes.

from xml.etree import ElementTree

with open('podcasts.opml', 'rt') as f:
    tree = ElementTree.parse(f)

for node in tree.getiterator('outline'):
    name = node.attrib.get('text')
    url = node.attrib.get('xmlUrl')
    if name and url:
        print '  %s :: %s' % (name, url)
    else:
        print name

Because we passed 'outline' to tree.getiterator() processing is
limited to only nodes with the tag 'outline'.

$ python ElementTree_show_feed_urls.py
Science and Tech
  APM: Future Tense :: http://www.publicradio.org/columns/futuretense/podcast.xml
  Engines Of Our Ingenuity Podcast :: http://www.npr.org/rss/podcast.php?id=510030
  Science & the City :: http://www.nyas.org/Podcasts/Atom.axd
Books and Fiction
  Podiobooker :: http://feeds.feedburner.com/podiobooks
  The Drabblecast :: http://web.me.com/normsherman/Site/Podcast/rss.xml
  tor.com / category / tordotstories :: http://www.tor.com/rss/category/TorDotStories
Computers and Programming
  MacBreak Weekly :: http://leo.am/podcasts/mbw
  FLOSS Weekly :: http://leo.am/podcasts/floss
  Core Intuition :: http://www.coreint.org/podcast.xml
Python
  PyCon Podcast :: http://advocacy.python.org/podcasts/pycon.rss
  A Little Bit of Python :: http://advocacy.python.org/podcasts/littlebit.rss
  Django Dose Everything Feed :: http://djangodose.com/everything/feed/
Miscelaneous
  dhellmann's CastSampler Feed :: http://www.castsampler.com/cast/feed/rss/dhellmann/

Finding Nodes in a Document

Walking the entire tree yourself like this searching for relevant
nodes can be error prone. In the example above, we had to look at
each outline node to determine if it was a group (nodes with only a
“text” attribute) or podcast (with both “text” and “xmlUrl”). If we
were writing a podcast downloader and needed to produce a simple list
of the podcast feed URLs, without names or groups, we might simplify
the logic using findall() to look for nodes with more descriptive
search characteristics.

A first pass at converting the above example might construct an XPath
argument to look for all outline nodes.

from xml.etree import ElementTree

with open('podcasts.opml', 'rt') as f:
    tree = ElementTree.parse(f)

for node in tree.findall('.//outline'):
    url = node.attrib.get('xmlUrl')
    if url:
        print url

The logic in this version is not substantially different than the
version using getiterator(). We still have to check for the
presence of the URL, except that we don’t print the group name when
the URL is not found.

$ python ElementTree_find_feeds_by_tag.py
http://www.publicradio.org/columns/futuretense/podcast.xml
http://www.npr.org/rss/podcast.php?id=510030
http://www.nyas.org/Podcasts/Atom.axd
http://feeds.feedburner.com/podiobooks
http://web.me.com/normsherman/Site/Podcast/rss.xml
http://www.tor.com/rss/category/TorDotStories
http://leo.am/podcasts/mbw
http://leo.am/podcasts/floss
http://www.coreint.org/podcast.xml
http://advocacy.python.org/podcasts/pycon.rss
http://advocacy.python.org/podcasts/littlebit.rss
http://djangodose.com/everything/feed/
http://www.castsampler.com/cast/feed/rss/dhellmann/

Another version can take advantage of the fact that we know the
outline nodes are only nested two levels deep. If we change the
search path to .//outline/outline we will process only the second
level of outline nodes.

from xml.etree import ElementTree

with open('podcasts.opml', 'rt') as f:
    tree = ElementTree.parse(f)

for node in tree.findall('.//outline/outline'):
    url = node.attrib.get('xmlUrl')
    print url

We expect all of those outline nodes nested 2 levels deep in the input
will have the xmlURL attribute refering to the podcast feed, so if we
were brave we could skip checking for for the attribute before using
it.

$ python ElementTree_find_feeds_by_structure.py
http://www.publicradio.org/columns/futuretense/podcast.xml
http://www.npr.org/rss/podcast.php?id=510030
http://www.nyas.org/Podcasts/Atom.axd
http://feeds.feedburner.com/podiobooks
http://web.me.com/normsherman/Site/Podcast/rss.xml
http://www.tor.com/rss/category/TorDotStories
http://leo.am/podcasts/mbw
http://leo.am/podcasts/floss
http://www.coreint.org/podcast.xml
http://advocacy.python.org/podcasts/pycon.rss
http://advocacy.python.org/podcasts/littlebit.rss
http://djangodose.com/everything/feed/
http://www.castsampler.com/cast/feed/rss/dhellmann/

This version is limited to our existing structure, though, so if the
outline nodes are ever rearranged into a deeper tree it will stop
working.

Parsed Node Attributes

The items returned by findall() and getiterator() are Element
objects, each representing a node in the XML parse tree. Each Element
has attributes for accessing data pulled out of the XML. This can be
illustrated with a somewhat more contrived example input file,
data.xml:

<?xml version="1.0" encoding="UTF-8"?>
<top>
  <child>This child contains text.</child>
  <child_with_tail>This child has regular text.</child_with_tail>And "tail" text.
  <with_attributes name="value" foo="bar" />
  <entity_expansion attribute="This &#38; That">That &#38; This</entity_expansion>
</top>

The “attributes” of a node are available in the attrib property,
which acts like a dictionary.

from xml.etree import ElementTree

with open('data.xml', 'rt') as f:
    tree = ElementTree.parse(f)

node = tree.find('./with_attributes')
print node.tag
for name, value in sorted(node.attrib.items()):
    print '  %-4s = "%s"' % (name, value)
    

The node on line 5 of the input file has 2 attributes, name and foo.

$ python ElementTree_node_attributes.py
with_attributes
  foo  = "bar"
  name = "value"

The text content of the nodes is available, along with the “tail” text
that comes after the end of a close tag.

from xml.etree import ElementTree

with open('data.xml', 'rt') as f:
    tree = ElementTree.parse(f)

for path in [ './child', './child_with_tail' ]:
    node = tree.find(path)
    print node.tag
    print '  child node text:', node.text
    print '  and tail text  :', node.tail

The child node on line 3 contains embedded text, and the node on
line 4 has text with a tail (including any whitespace).

$ python ElementTree_node_text.py
child
  child node text: This child contains text.
  and tail text  :

child_with_tail
  child node text: This child has regular text.
  and tail text  : And "tail" text.

Conveniently, XML entity references embedded in the document are
converted to the appropriate characters before values are returned.

from xml.etree import ElementTree

with open('data.xml', 'rt') as f:
    tree = ElementTree.parse(f)

node = tree.find('entity_expansion')
print node.tag
print '  in attribute:', node.attrib['attribute']
print '  in text     :', node.text

The conversion saves you from having to worry about an implementation
detail of representing certain characters in an XML document.

$ python ElementTree_entity_references.py
entity_expansion
  in attribute: This & That
  in text     : That & This

Watching Events While Parsing

The other API useful for processing XML documents is event-based. The
parser generates start events for opening tags and end events
for closing tags. Iterating over the event stream lets you extract
data from the document while parsing it, which is convenient if you
don’t need to manipulate the entire document afterwards and if you
want to avoid holding the entire parsed document in memory.

iterparse() returns an iterable that produces tuples containing
the name of the event and the node triggering the event. Events can
be one of:

start: A new tag has been encountered. The closing angle bracket of the
tag was processed, but not the contents.
end: The closing angle bracket of a closing tag has been processed. All
of the children were already processed.
start-ns: Start a namespace declaration.
end-ns: End a namespace declaration.

from xml.etree.ElementTree import iterparse

depth = 0
prefix_width = 8
prefix_dots = '.' * prefix_width
line_template = '{prefix:<0.{prefix_len}}{event:<8}{suffix:<{suffix_len}} {node.tag:<12} {node_id}'

for (event, node) in iterparse('podcasts.opml', ['start', 'end', 'start-ns', 'end-ns']):
    if event == 'end':
        depth -= 1

    prefix_len = depth * 2
    
    print line_template.format(prefix=prefix_dots,
                               prefix_len=prefix_len,
                               suffix='',
                               suffix_len=(prefix_width - prefix_len),
                               node=node,
                               node_id=id(node),
                               event=event,
                               )
    
    if event == 'start':
        depth += 1

By default, only end events are generated. To see other events,
pass the list of event names you want to receive to iterparse(),
as in this example:

$ python ElementTree_show_all_events.py
start            opml         876256
..start          head         876336
....start        title        888920
....end          title        888920
....start        dateCreated  889280
....end          dateCreated  889280
....start        dateModified 889320
....end          dateModified 889320
..end            head         876336
..start          body         889400
....start        outline      889560
......start      outline      889600
......end        outline      889600
......start      outline      889480
......end        outline      889480
......start      outline      889680
......end        outline      889680
....end          outline      889560
....start        outline      889720
......start      outline      889760
......end        outline      889760
......start      outline      889840
......end        outline      889840
......start      outline      889920
......end        outline      889920
....end          outline      889720
....start        outline      889880
......start      outline      890040
......end        outline      890040
......start      outline      890120
......end        outline      890120
......start      outline      890200
......end        outline      890200
....end          outline      889880
....start        outline      890240
......start      outline      890360
......end        outline      890360
......start      outline      890440
......end        outline      890440
......start      outline      890520
......end        outline      890520
....end          outline      890240
....start        outline      890640
......start      outline      890720
......end        outline      890720
....end          outline      890640
..end            body         889400
end              opml         876256

The event-style of processing may be more natural for some operations,
such as converting XML input to some other format. For example,
suppose we want to convert the list of podcasts we have been working
with from an XML file to a data file we can load into a spreadsheet or
database application. We don’t need to hold the entire data set in
memory at a time, since we’re simply changing the format.

import csv
from xml.etree.ElementTree import iterparse
import sys

writer = csv.writer(sys.stdout, quoting=csv.QUOTE_NONNUMERIC)

group_name = ''

for (event, node) in iterparse('podcasts.opml', events=['start']):
    if node.tag != 'outline':
        # Ignore anything not part of the outline
        continue
    if not node.attrib.get('xmlUrl'):
        # Remember the current group
        group_name = node.attrib['text']
    else:
        # Output a podcast entry
        writer.writerow( (group_name, node.attrib['text'],
                          node.attrib['xmlUrl'],
                          node.attrib.get('htmlUrl', ''),
                          )
                         )

This example program converts our podcast list to a CSV file, ready to
be imported into another application.

$ python ElementTree_write_podcast_csv.py
"Science and Tech","APM: Future Tense","http://www.publicradio.org/columns/futuretense/podcast.xml","http://www.publicradio.org/columns/futuretense/"
"Science and Tech","Engines Of Our Ingenuity Podcast","http://www.npr.org/rss/podcast.php?id=510030","http://www.uh.edu/engines/engines.htm"
"Science and Tech","Science & the City","http://www.nyas.org/Podcasts/Atom.axd","http://www.nyas.org/WhatWeDo/SciencetheCity.aspx"
"Books and Fiction","Podiobooker","http://feeds.feedburner.com/podiobooks","http://www.podiobooks.com/blog"
"Books and Fiction","The Drabblecast","http://web.me.com/normsherman/Site/Podcast/rss.xml","http://web.me.com/normsherman/Site/Podcast/Podcast.html"
"Books and Fiction","tor.com / category / tordotstories","http://www.tor.com/rss/category/TorDotStories","http://www.tor.com/"
"Computers and Programming","MacBreak Weekly","http://leo.am/podcasts/mbw","http://twit.tv/mbw"
"Computers and Programming","FLOSS Weekly","http://leo.am/podcasts/floss","http://twit.tv"
"Computers and Programming","Core Intuition","http://www.coreint.org/podcast.xml","http://www.coreint.org/"
"Python","PyCon Podcast","http://advocacy.python.org/podcasts/pycon.rss","http://advocacy.python.org/podcasts/"
"Python","A Little Bit of Python","http://advocacy.python.org/podcasts/littlebit.rss","http://advocacy.python.org/podcasts/"
"Python","Django Dose Everything Feed","http://djangodose.com/everything/feed/",""
"Miscelaneous","dhellmann's CastSampler Feed","http://www.castsampler.com/cast/feed/rss/dhellmann/","http://www.castsampler.com/users/dhellmann/"

Creating Your Own Tree Builder

A potentially more efficient means of handling parse events is to
replace the standard tree builder behavior with your own. The
ElementTree parser uses an XMLTreeBuilder to process the XML and call
methods on a target class to save the results. The usual output is an
ElementTree instance created by the default TreeBuilder class. By
replacing TreeBuilder with your own class, you can receive the events
before the Element nodes are instantiated, saving that portion of the
overhead.

The XML-to-CSV app from the previous section can be translated to a
tree builder.

import csv
from xml.etree.ElementTree import XMLTreeBuilder
import sys

class PodcastListToCSV(object):

    def __init__(self, outputFile):
        self.writer = csv.writer(outputFile, quoting=csv.QUOTE_NONNUMERIC)
        self.group_name = ''
        return

    def start(self, tag, attrib):
        if tag != 'outline':
            # Ignore anything not part of the outline
            return
        if not attrib.get('xmlUrl'):
            # Remember the current group
            self.group_name = attrib['text']
        else:
            # Output a podcast entry
            self.writer.writerow( (self.group_name, attrib['text'],
                                   attrib['xmlUrl'],
                                   attrib.get('htmlUrl', ''),
                                   )
                                  )

    def end(self, tag):
        # Ignore closing tags
        pass
    def data(self, data):
        # Ignore data inside nodes
        pass
    def close(self):
        # Nothing special to do here
        return


target = PodcastListToCSV(sys.stdout)
parser = XMLTreeBuilder(target=target)
with open('podcasts.opml', 'rt') as f:
    for line in f:
        parser.feed(line)
parser.close()

PodcastListToCSV implements the TreeBuilder protocol. Each time a
new XML tag is encountered, start() is called with the tag name
and attributes. When a closing tag is seen end() is called with
the name. In between, data() is called when a node has content
(the tree builder is expected to keep up with the “current” node).
When all of the input is processed, close() is called. It can
return a value, which will be returned to the user of the
XMLTreeBuilder.

$ python ElementTree_podcast_csv_treebuilder.py
"Science and Tech","APM: Future Tense","http://www.publicradio.org/columns/futuretense/podcast.xml","http://www.publicradio.org/columns/futuretense/"
"Science and Tech","Engines Of Our Ingenuity Podcast","http://www.npr.org/rss/podcast.php?id=510030","http://www.uh.edu/engines/engines.htm"
"Science and Tech","Science & the City","http://www.nyas.org/Podcasts/Atom.axd","http://www.nyas.org/WhatWeDo/SciencetheCity.aspx"
"Books and Fiction","Podiobooker","http://feeds.feedburner.com/podiobooks","http://www.podiobooks.com/blog"
"Books and Fiction","The Drabblecast","http://web.me.com/normsherman/Site/Podcast/rss.xml","http://web.me.com/normsherman/Site/Podcast/Podcast.html"
"Books and Fiction","tor.com / category / tordotstories","http://www.tor.com/rss/category/TorDotStories","http://www.tor.com/"
"Computers and Programming","MacBreak Weekly","http://leo.am/podcasts/mbw","http://twit.tv/mbw"
"Computers and Programming","FLOSS Weekly","http://leo.am/podcasts/floss","http://twit.tv"
"Computers and Programming","Core Intuition","http://www.coreint.org/podcast.xml","http://www.coreint.org/"
"Python","PyCon Podcast","http://advocacy.python.org/podcasts/pycon.rss","http://advocacy.python.org/podcasts/"
"Python","A Little Bit of Python","http://advocacy.python.org/podcasts/littlebit.rss","http://advocacy.python.org/podcasts/"
"Python","Django Dose Everything Feed","http://djangodose.com/everything/feed/",""
"Miscelaneous","dhellmann's CastSampler Feed","http://www.castsampler.com/cast/feed/rss/dhellmann/","http://www.castsampler.com/users/dhellmann/"

Parsing Strings

To work with smaller bits of XML text, especially string literals as
might be embedded in the source of a program, use
xml.etree.ElementTree.XML and pass a single argument, the string
containing the XML to be parsed.

from xml.etree.ElementTree import XML

parsed = XML('''
<root>
  <group>
    <child id="a">This is child "a".</child>
    <child id="b">This is child "b".</child>
  </group>
  <group>
    <child id="c">This is child "c".</child>
  </group>
</root>
''')

print 'parsed =', parsed

for elem in parsed.getiterator():
    print elem.tag
    if elem.text is not None and elem.text.strip():
        print '  text: "%s"' % elem.text
    if elem.tail is not None and elem.tail.strip():
        print '  tail: "%s"' % elem.tail
    for name, value in sorted(elem.attrib.items()):
        print '  %-4s = "%s"' % (name, value)
    print

Notice that unlike with parse(), the return value is an Element
instance instead of an ElementTree.

$ python ElementTree_XML.py
parsed = <Element root at d4e40>
root

group

child
  text: "This is child "a"."
  id   = "a"

child
  text: "This is child "b"."
  id   = "b"

group

child
  text: "This is child "c"."
  id   = "c"

For structured XML that uses the “id” attribute to identify unique
nodes of interest, XMLID() is a convenient way to access the parse
results.

from xml.etree.ElementTree import XMLID

tree, id_map = XMLID('''
<root>
  <group>
    <child id="a">This is child "a".</child>
    <child id="b">This is child "b".</child>
  </group>
  <group>
    <child id="c">This is child "c".</child>
  </group>
</root>
''')

for key, value in sorted(id_map.items()):
    print '%s = %s' % (key, value)
    

XMLID() returns the parsed tree as an Element object,
along with a dictionary mapping the id attribute strings to the
individual nodes in the tree.

$ python ElementTree_XMLID.py
a = <Element child at d3eb8>
b = <Element child at d3d78>
c = <Element child at d9030>

See also

Outline Processor Markup Language, OPML: Dave Winer’s OPML specification and documentation.
XPath Support in ElementTree: Part of Fredrick Lundh’s original documentation for ElementTree.
csv: Read and write comma-separated-value files

PyMOTW Home

The canonical version of this article

PyMOTW: tabnanny - Indentation validator

2010-03-07T09:38:00.001-05:00

tabnanny – Indentation validator

Purpose:	Scan Python source code looking for suspicious indentation.
Python Version:	2.1.3 and later

Consistent use of indentation is important in a langauge like Python,
where white-space is significant. The tabnanny module provides a
scanner to report on “ambiguous” use of indentation.

Running from the Command Line

The simplest way to use tabnanny is to run it from the command line,
passing the names of files to check. If you pass directory names, the
directories are scanned recursively to find .py files to check.

When I ran tabnanny across the PyMOTW source code, I found one old
module with tabs instead of spaces:

$ python -m tabnanny .
./PyMOTW/Queue/fetch_podcasts.py 78 "\t\tfor enclosure in entry.get('enclosures', []):\n"

Sure enough, line 78 of fetch_podcasts.py had two tabs instead of 8
spaces. I didn’t see this by looking at it in my editor because I
have my tabstops set to 4 spaces, so visually there was no difference.

for enclosure in entry.get('enclosures', []):
    print 'Queuing:', enclosure['url']
    enclosure_queue.put(enclosure['url'])

Correcting line 78 and running tabnanny again showed another error on
line 79. One last problem showed up on line 80.

If you want to scan files, but not see the details about the error,
you can use the -q option to suppress all information except the
filename.

$ python -m tabnanny -q .
./PyMOTW/Queue/fetch_podcasts.py

To see more information about the files being scanned, use the -v option.

$ python -m tabnanny -v ./PyMOTW/Queue
'./PyMOTW/Queue': listing directory
'./PyMOTW/Queue/__init__.py': Clean bill of health.
'./PyMOTW/Queue/feedparser.py': Clean bill of health.
'./PyMOTW/Queue/fetch_podcasts.py': *** Line 78: trouble in tab city! ***
offending line: "\t\tfor enclosure in entry.get('enclosures', []):\n"
indent not greater e.g. at tab sizes 1, 2

Using within Your Program

As soon as I discovered the mistake in my Queue example, I decided I
needed to add a tabnanny check to my PyMOTW build process. I created
a tabcheck task in my pavement.py build script so I could run paver
tabcheck and scan the code I’m working on for PyMOTW. This is
possible because tabnanny exposes its check() function as a public
API.

Here’s an example of using tabnanny that doesn’t require understanding
Paver’s task definition decorators.

import sys
import tabnanny

# Turn on verbose mode
tabnanny.verbose = 1

for dirname in sys.argv[1:]:
    tabnanny.check(dirname)

And in action:

$ python tabnanny_check.py ../Queue
'../Queue': listing directory
'../Queue/__init__.py': Clean bill of health.
'../Queue/feedparser.py': Clean bill of health.
'../Queue/fetch_podcasts.py': *** Line 78: trouble in tab city! ***
offending line: "\t\tfor enclosure in entry.get('enclosures', []):\n"
indent not greater e.g. at tab sizes 1, 2

Note

If you run these examples against the PyMOTW code it won’t report
the same errors, since I have fixed my code in this release.

See also

tabnanny: The standard library documentation for this module.
tokenize: Lexical scanner for Python source code.

PyMOTW Home

The canonical version of this article

PyMOTW: cgitb - Detailed traceback reports

2010-01-30T13:11:00.001-05:00

cgitb – Detailed traceback reports

Purpose:	cgitb provides more detailed traceback information than `traceback`.
Python Version:	2.2 and later

cgitb was originally designed for showing errors and debugging
information in web applications. It was later updated to include
plain-text output as well, but unfortunately wasn’t renamed. This has
led to obscurity and the module is not used as often as it should be.
Nonetheless, cgitb is a valuable debugging tool in the standard
library.

Standard Traceback Dumps

Python’s default exception handling behavior is to print a tracebackto standard error with the call stack leading up to the error
position. This basic output is frequently enough information to
understand the cause of the exception and permit a fix.

def func1(arg1):
    local_var = arg1 * 2
    return func2(local_var)

def func2(arg2):
    local_var = arg2 + 2
    return func3(local_var)

def func3(arg3):
    local_var = arg2 / 2
    return local_var

func1(1)

The above sample program has a subtle error in func3().

$ python cgitb_basic_traceback.py
Traceback (most recent call last):
  File "cgitb_basic_traceback.py", line 22, in <module>
    func1(1)
  File "cgitb_basic_traceback.py", line 12, in func1
    return func2(local_var)
  File "cgitb_basic_traceback.py", line 16, in func2
    return func3(local_var)
  File "cgitb_basic_traceback.py", line 19, in func3
    local_var = arg2 / 2
NameError: global name 'arg2' is not defined

Enabling Detailed Tracebacks

While the basic traceback includes enough information for us to spot
the error, enabling cgitb replaces sys.excepthook with a function
that gives extended tracebacks with even more detail.

import cgitb
cgitb.enable(format='text')

def func1(arg1):
    local_var = arg1 * 2
    return func2(local_var)

def func2(arg2):
    local_var = arg2 + 2
    return func3(local_var)

def func3(arg3):
    local_var = arg2 / 2
    return local_var

func1(1)

As you can see below, the error report is much more extensive. Each
frame of the stack is listed, along with:

the full path to the source file, instead of just the base name
the values of the arguments to each function in the stack
a few lines of source context from around the line in the error path
the values of variables in the expression causing the error

$ python cgitb_extended_traceback.py
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/cgitb.py:245: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  value = pydoc.text.repr(getattr(evalue, name))
<type 'exceptions.NameError'>
Python 2.6.2: /Users/dhellmann/.virtualenvs/pymotw/bin/python
Sat Jan 30 12:50:27 2010

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_extended_traceback.py in <module>()
   21 def func3(arg3):
   22     local_var = arg2 / 2
   23     return local_var
   24
   25 func1(1)
func1 = <function func1 at 0x7bbb0>

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_extended_traceback.py in func1(arg1=1)
   13 def func1(arg1):
   14     local_var = arg1 * 2
   15     return func2(local_var)
   16
   17 def func2(arg2):
global func2 = <function func2 at 0x7bbf0>
local_var = 2

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_extended_traceback.py in func2(arg2=2)
   17 def func2(arg2):
   18     local_var = arg2 + 2
   19     return func3(local_var)
   20
   21 def func3(arg3):
global func3 = <function func3 at 0x7bc30>
local_var = 4

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_extended_traceback.py in func3(arg3=4)
   20
   21 def func3(arg3):
   22     local_var = arg2 / 2
   23     return local_var
   24
local_var undefined
arg2 undefined
<type 'exceptions.NameError'>: global name 'arg2' is not defined
    __class__ = <type 'exceptions.NameError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.NameError object at 0x81710>
    __dict__ = {}
    __doc__ = 'Name not found globally.'
    __format__ = <built-in method __format__ of exceptions.NameError object at 0x81710>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.NameError object at 0x81710>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.NameError object at 0x81710>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.NameError object at 0x81710>
    __hash__ = <method-wrapper '__hash__' of exceptions.NameError object at 0x81710>
    __init__ = <method-wrapper '__init__' of exceptions.NameError object at 0x81710>
    __new__ = <built-in method __new__ of type object at 0x5b0aa0>
    __reduce__ = <built-in method __reduce__ of exceptions.NameError object at 0x81710>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.NameError object at 0x81710>
    __repr__ = <method-wrapper '__repr__' of exceptions.NameError object at 0x81710>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.NameError object at 0x81710>
    __setstate__ = <built-in method __setstate__ of exceptions.NameError object at 0x81710>
    __sizeof__ = <built-in method __sizeof__ of exceptions.NameError object at 0x81710>
    __str__ = <method-wrapper '__str__' of exceptions.NameError object at 0x81710>
    __subclasshook__ = <built-in method __subclasshook__ of type object at 0x5b0aa0>
    __unicode__ = <built-in method __unicode__ of exceptions.NameError object at 0x81710>
    args = ("global name 'arg2' is not defined",)
    message = "global name 'arg2' is not defined"

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "cgitb_extended_traceback.py", line 25, in <module>
    func1(1)
  File "cgitb_extended_traceback.py", line 15, in func1
    return func2(local_var)
  File "cgitb_extended_traceback.py", line 19, in func2
    return func3(local_var)
  File "cgitb_extended_traceback.py", line 22, in func3
    local_var = arg2 / 2
NameError: global name 'arg2' is not defined

The end of the output also includes the full details of the exception
object (in case it has attributes other than message that would be
useful for debugging) and the original form of a traceback dump.

Local Variables in Tracebacks

Having access to the variables involved in the error stack can help
find a logical error that occurs somewhere higher in the stack than
the line where the actual exception is generated.

import cgitb
cgitb.enable(format='text')

def func2(a, divisor):
    return a / divisor

def func1(a, b):
    c = b - 5
    return func2(a, c)

func1(1, 5)

In the case of this code with a ZeroDivisionError, we can see that the
problem is introduced in the computation of the value of c in
func1(), rather than where the value is used in func2().

$ python cgitb_local_vars.py
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/cgitb.py:245: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  value = pydoc.text.repr(getattr(evalue, name))
<type 'exceptions.ZeroDivisionError'>
Python 2.6.2: /Users/dhellmann/.virtualenvs/pymotw/bin/python
Sat Jan 30 12:50:27 2010

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_local_vars.py in <module>()
   16 def func1(a, b):
   17     c = b - 5
   18     return func2(a, c)
   19
   20 func1(1, 5)
func1 = <function func1 at 0x7bbf0>

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_local_vars.py in func1(a=1, b=5)
   16 def func1(a, b):
   17     c = b - 5
   18     return func2(a, c)
   19
   20 func1(1, 5)
global func2 = <function func2 at 0x7bbb0>
a = 1
c = 0

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_local_vars.py in func2(a=1, divisor=0)
   12
   13 def func2(a, divisor):
   14     return a / divisor
   15
   16 def func1(a, b):
a = 1
divisor = 0
<type 'exceptions.ZeroDivisionError'>: integer division or modulo by zero
    __class__ = <type 'exceptions.ZeroDivisionError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.ZeroDivisionError object at 0x825a8>
    __dict__ = {}
    __doc__ = 'Second argument to a division or modulo operation was zero.'
    __format__ = <built-in method __format__ of exceptions.ZeroDivisionError object at 0x825a8>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.ZeroDivisionError object at 0x825a8>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.ZeroDivisionError object at 0x825a8>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.ZeroDivisionError object at 0x825a8>
    __hash__ = <method-wrapper '__hash__' of exceptions.ZeroDivisionError object at 0x825a8>
    __init__ = <method-wrapper '__init__' of exceptions.ZeroDivisionError object at 0x825a8>
    __new__ = <built-in method __new__ of type object at 0x5af9c0>
    __reduce__ = <built-in method __reduce__ of exceptions.ZeroDivisionError object at 0x825a8>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.ZeroDivisionError object at 0x825a8>
    __repr__ = <method-wrapper '__repr__' of exceptions.ZeroDivisionError object at 0x825a8>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.ZeroDivisionError object at 0x825a8>
    __setstate__ = <built-in method __setstate__ of exceptions.ZeroDivisionError object at 0x825a8>
    __sizeof__ = <built-in method __sizeof__ of exceptions.ZeroDivisionError object at 0x825a8>
    __str__ = <method-wrapper '__str__' of exceptions.ZeroDivisionError object at 0x825a8>
    __subclasshook__ = <built-in method __subclasshook__ of type object at 0x5af9c0>
    __unicode__ = <built-in method __unicode__ of exceptions.ZeroDivisionError object at 0x825a8>
    args = ('integer division or modulo by zero',)
    message = 'integer division or modulo by zero'

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "cgitb_local_vars.py", line 20, in <module>
    func1(1, 5)
  File "cgitb_local_vars.py", line 18, in func1
    return func2(a, c)
  File "cgitb_local_vars.py", line 14, in func2
    return a / divisor
ZeroDivisionError: integer division or modulo by zero

The code in cgitb that examines the variables used in the stack frame
leading to the error is smart enough to evaluate object attributes to
display them, too.

import cgitb
cgitb.enable(format='text')

class BrokenClass(object):
    """This class has an error.
    """
    
    def __init__(self, a, b):
        """Be careful passing arguments in here.
        """
        self.a = a
        self.b = b
        self.c = self.a * self.b
        self.d = self.a / self.b
        return

o = BrokenClass(1, 0)

Here we see that self.a and self.b are involved in the
error-prone code.

$ python cgitb_with_classes.py
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/cgitb.py:245: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  value = pydoc.text.repr(getattr(evalue, name))
<type 'exceptions.ZeroDivisionError'>
Python 2.6.2: /Users/dhellmann/.virtualenvs/pymotw/bin/python
Sat Jan 30 12:50:27 2010

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_with_classes.py in <module>()
   24         return
   25
   26 o = BrokenClass(1, 0)
   27
   28
o undefined
BrokenClass = <class '__main__.BrokenClass'>

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_with_classes.py in __init__(self=<__main__.BrokenClass object at 0x81e30>, a=1, b=0)
   21         self.b = b
   22         self.c = self.a * self.b
   23         self.d = self.a / self.b
   24         return
   25
self = <__main__.BrokenClass object at 0x81e30>
self.d undefined
self.a = 1
self.b = 0
<type 'exceptions.ZeroDivisionError'>: integer division or modulo by zero
    __class__ = <type 'exceptions.ZeroDivisionError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.ZeroDivisionError object at 0x7f710>
    __dict__ = {}
    __doc__ = 'Second argument to a division or modulo operation was zero.'
    __format__ = <built-in method __format__ of exceptions.ZeroDivisionError object at 0x7f710>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.ZeroDivisionError object at 0x7f710>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.ZeroDivisionError object at 0x7f710>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.ZeroDivisionError object at 0x7f710>
    __hash__ = <method-wrapper '__hash__' of exceptions.ZeroDivisionError object at 0x7f710>
    __init__ = <method-wrapper '__init__' of exceptions.ZeroDivisionError object at 0x7f710>
    __new__ = <built-in method __new__ of type object at 0x5af9c0>
    __reduce__ = <built-in method __reduce__ of exceptions.ZeroDivisionError object at 0x7f710>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.ZeroDivisionError object at 0x7f710>
    __repr__ = <method-wrapper '__repr__' of exceptions.ZeroDivisionError object at 0x7f710>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.ZeroDivisionError object at 0x7f710>
    __setstate__ = <built-in method __setstate__ of exceptions.ZeroDivisionError object at 0x7f710>
    __sizeof__ = <built-in method __sizeof__ of exceptions.ZeroDivisionError object at 0x7f710>
    __str__ = <method-wrapper '__str__' of exceptions.ZeroDivisionError object at 0x7f710>
    __subclasshook__ = <built-in method __subclasshook__ of type object at 0x5af9c0>
    __unicode__ = <built-in method __unicode__ of exceptions.ZeroDivisionError object at 0x7f710>
    args = ('integer division or modulo by zero',)
    message = 'integer division or modulo by zero'

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "cgitb_with_classes.py", line 26, in <module>
    o = BrokenClass(1, 0)
  File "cgitb_with_classes.py", line 23, in __init__
    self.d = self.a / self.b
ZeroDivisionError: integer division or modulo by zero

Adding More Context

Suppose your function includes a lot of inline comments, whitespace,
or other code that makes it very long. Having the default of 5 lines
of context may not be enough help in that case, if the body of the
function is pushed out of the code window displayed. Using a larger
context value when enabling cgitb gets around this.

import cgitb
import sys

context_length = int(sys.argv[1])
cgitb.enable(format='text', context=context_length)

def func2(a, divisor):
    return a / divisor

def func1(a, b):
    c = b - 5
    # Really
    # long
    # comment
    # goes
    # here.
    return func2(a, c)

func1(1, 5)

You can pass context to cgitb.enable() to control the amount
of code displayed for each line of the traceback.

$ python cgitb_more_context.py 5
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/cgitb.py:245: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  value = pydoc.text.repr(getattr(evalue, name))
<type 'exceptions.ZeroDivisionError'>
Python 2.6.2: /Users/dhellmann/.virtualenvs/pymotw/bin/python
Sat Jan 30 12:50:27 2010

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_more_context.py in <module>()
   24     # goes
   25     # here.
   26     return func2(a, c)
   27
   28 func1(1, 5)
func1 = <function func1 at 0x7bbb0>

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_more_context.py in func1(a=1, b=5)
   24     # goes
   25     # here.
   26     return func2(a, c)
   27
   28 func1(1, 5)
global func2 = <function func2 at 0x7bb70>
a = 1
c = 0

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_more_context.py in func2(a=1, divisor=0)
   15
   16 def func2(a, divisor):
   17     return a / divisor
   18
   19 def func1(a, b):
a = 1
divisor = 0
<type 'exceptions.ZeroDivisionError'>: integer division or modulo by zero
    __class__ = <type 'exceptions.ZeroDivisionError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.ZeroDivisionError object at 0x816c0>
    __dict__ = {}
    __doc__ = 'Second argument to a division or modulo operation was zero.'
    __format__ = <built-in method __format__ of exceptions.ZeroDivisionError object at 0x816c0>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.ZeroDivisionError object at 0x816c0>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.ZeroDivisionError object at 0x816c0>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.ZeroDivisionError object at 0x816c0>
    __hash__ = <method-wrapper '__hash__' of exceptions.ZeroDivisionError object at 0x816c0>
    __init__ = <method-wrapper '__init__' of exceptions.ZeroDivisionError object at 0x816c0>
    __new__ = <built-in method __new__ of type object at 0x5af9c0>
    __reduce__ = <built-in method __reduce__ of exceptions.ZeroDivisionError object at 0x816c0>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.ZeroDivisionError object at 0x816c0>
    __repr__ = <method-wrapper '__repr__' of exceptions.ZeroDivisionError object at 0x816c0>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.ZeroDivisionError object at 0x816c0>
    __setstate__ = <built-in method __setstate__ of exceptions.ZeroDivisionError object at 0x816c0>
    __sizeof__ = <built-in method __sizeof__ of exceptions.ZeroDivisionError object at 0x816c0>
    __str__ = <method-wrapper '__str__' of exceptions.ZeroDivisionError object at 0x816c0>
    __subclasshook__ = <built-in method __subclasshook__ of type object at 0x5af9c0>
    __unicode__ = <built-in method __unicode__ of exceptions.ZeroDivisionError object at 0x816c0>
    args = ('integer division or modulo by zero',)
    message = 'integer division or modulo by zero'

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "cgitb_more_context.py", line 28, in <module>
    func1(1, 5)
  File "cgitb_more_context.py", line 26, in func1
    return func2(a, c)
  File "cgitb_more_context.py", line 17, in func2
    return a / divisor
ZeroDivisionError: integer division or modulo by zero

Increasing the value gets us enough of the function that we can spot
the problem in the code, again.

$ python cgitb_more_context.py 10
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/cgitb.py:245: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  value = pydoc.text.repr(getattr(evalue, name))
<type 'exceptions.ZeroDivisionError'>
Python 2.6.2: /Users/dhellmann/.virtualenvs/pymotw/bin/python
Sat Jan 30 12:50:28 2010

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_more_context.py in <module>()
   19 def func1(a, b):
   20     c = b - 5
   21     # Really
   22     # long
   23     # comment
   24     # goes
   25     # here.
   26     return func2(a, c)
   27
   28 func1(1, 5)
func1 = <function func1 at 0x7bbb0>

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_more_context.py in func1(a=1, b=5)
   19 def func1(a, b):
   20     c = b - 5
   21     # Really
   22     # long
   23     # comment
   24     # goes
   25     # here.
   26     return func2(a, c)
   27
   28 func1(1, 5)
global func2 = <function func2 at 0x7bb70>
a = 1
c = 0

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_more_context.py in func2(a=1, divisor=0)
   12
   13 context_length = int(sys.argv[1])
   14 cgitb.enable(format='text', context=context_length)
   15
   16 def func2(a, divisor):
   17     return a / divisor
   18
   19 def func1(a, b):
   20     c = b - 5
   21     # Really
a = 1
divisor = 0
<type 'exceptions.ZeroDivisionError'>: integer division or modulo by zero
    __class__ = <type 'exceptions.ZeroDivisionError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.ZeroDivisionError object at 0x816c0>
    __dict__ = {}
    __doc__ = 'Second argument to a division or modulo operation was zero.'
    __format__ = <built-in method __format__ of exceptions.ZeroDivisionError object at 0x816c0>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.ZeroDivisionError object at 0x816c0>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.ZeroDivisionError object at 0x816c0>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.ZeroDivisionError object at 0x816c0>
    __hash__ = <method-wrapper '__hash__' of exceptions.ZeroDivisionError object at 0x816c0>
    __init__ = <method-wrapper '__init__' of exceptions.ZeroDivisionError object at 0x816c0>
    __new__ = <built-in method __new__ of type object at 0x5af9c0>
    __reduce__ = <built-in method __reduce__ of exceptions.ZeroDivisionError object at 0x816c0>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.ZeroDivisionError object at 0x816c0>
    __repr__ = <method-wrapper '__repr__' of exceptions.ZeroDivisionError object at 0x816c0>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.ZeroDivisionError object at 0x816c0>
    __setstate__ = <built-in method __setstate__ of exceptions.ZeroDivisionError object at 0x816c0>
    __sizeof__ = <built-in method __sizeof__ of exceptions.ZeroDivisionError object at 0x816c0>
    __str__ = <method-wrapper '__str__' of exceptions.ZeroDivisionError object at 0x816c0>
    __subclasshook__ = <built-in method __subclasshook__ of type object at 0x5af9c0>
    __unicode__ = <built-in method __unicode__ of exceptions.ZeroDivisionError object at 0x816c0>
    args = ('integer division or modulo by zero',)
    message = 'integer division or modulo by zero'

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "cgitb_more_context.py", line 28, in <module>
    func1(1, 5)
  File "cgitb_more_context.py", line 26, in func1
    return func2(a, c)
  File "cgitb_more_context.py", line 17, in func2
    return a / divisor
ZeroDivisionError: integer division or modulo by zero

Exception Properties

In addition to the local variables from each stack frame, cgitb shows
all properties of the exception object. If you have a custom
exception type with extra properties, they are printed as part of the
error report.

import cgitb
cgitb.enable(format='text')

class MyException(Exception):
    """Add extra properties to a special exception
    """
    
    def __init__(self, message, bad_value):
        self.bad_value = bad_value
        Exception.__init__(self, message)
        return

raise MyException('Normal message', bad_value=99)

In this example, the bad_value property is included along with the
standard message and args values.

$ python cgitb_exception_properties.py
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/cgitb.py:245: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  value = pydoc.text.repr(getattr(evalue, name))
<class '__main__.MyException'>
Python 2.6.2: /Users/dhellmann/.virtualenvs/pymotw/bin/python
Sat Jan 30 12:50:28 2010

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_exception_properties.py in <module>()
   18         self.bad_value = bad_value
   19         Exception.__init__(self, message)
   20         return
   21
   22 raise MyException('Normal message', bad_value=99)
MyException = <class '__main__.MyException'>
bad_value undefined
<class '__main__.MyException'>: Normal message
    __class__ = <class '__main__.MyException'>
    __delattr__ = <method-wrapper '__delattr__' of MyException object at 0x7f698>
    __dict__ = {'bad_value': 99}
    __doc__ = 'Add extra properties to a special exception\n    '
    __format__ = <built-in method __format__ of MyException object at 0x7f698>
    __getattribute__ = <method-wrapper '__getattribute__' of MyException object at 0x7f698>
    __getitem__ = <method-wrapper '__getitem__' of MyException object at 0x7f698>
    __getslice__ = <method-wrapper '__getslice__' of MyException object at 0x7f698>
    __hash__ = <method-wrapper '__hash__' of MyException object at 0x7f698>
    __init__ = <bound method MyException.__init__ of MyException('Normal message',)>
    __module__ = '__main__'
    __new__ = <built-in method __new__ of type object at 0x5aebc0>
    __reduce__ = <built-in method __reduce__ of MyException object at 0x7f698>
    __reduce_ex__ = <built-in method __reduce_ex__ of MyException object at 0x7f698>
    __repr__ = <method-wrapper '__repr__' of MyException object at 0x7f698>
    __setattr__ = <method-wrapper '__setattr__' of MyException object at 0x7f698>
    __setstate__ = <built-in method __setstate__ of MyException object at 0x7f698>
    __sizeof__ = <built-in method __sizeof__ of MyException object at 0x7f698>
    __str__ = <method-wrapper '__str__' of MyException object at 0x7f698>
    __subclasshook__ = <built-in method __subclasshook__ of type object at 0xa16890>
    __unicode__ = <built-in method __unicode__ of MyException object at 0x7f698>
    __weakref__ = None
    args = ('Normal message',)
    bad_value = 99
    message = 'Normal message'

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "cgitb_exception_properties.py", line 22, in <module>
    raise MyException('Normal message', bad_value=99)
MyException: Normal message

Logging Tracebacks

For many situations, printing the traceback details to standard error
is the best resolution. In a production system, however, logging the
errors is even better. cgitb.enable() includes an optional
argument, logdir, to enable error logging. When a directory name
is provided, each exception is logged to its own file in the given
directory.

import cgitb
import os

cgitb.enable(logdir=os.path.join(os.path.dirname(__file__), 'LOGS'),
             display=False,
             format='text',
             )

def func2(a, divisor):
    return a / divisor

def func1(a, b):
    c = b - 5
    return func2(a, c)

func1(1, 5)

Event with the error display suppressed, a message is printed
describing where to go to find the error log.

$ python cgitb_log_exception.py
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/cgitb.py:245: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  value = pydoc.text.repr(getattr(evalue, name))
<p>A problem occurred in a Python script.
<p> /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/LOGS/tmpohlTP8.txt contains the description of this error.

$ ls LOGS
tmpohlTP8.txt

$ cat LOGS/*.txt
<type 'exceptions.ZeroDivisionError'>
Python 2.6.2: /Users/dhellmann/.virtualenvs/pymotw/bin/python
Sat Jan 30 12:50:28 2010

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_log_exception.py in <module>()
   21 def func1(a, b):
   22     c = b - 5
   23     return func2(a, c)
   24
   25 func1(1, 5)
func1 = <function func1 at 0x7bbb0>

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_log_exception.py in func1(a=1, b=5)
   21 def func1(a, b):
   22     c = b - 5
   23     return func2(a, c)
   24
   25 func1(1, 5)
global func2 = <function func2 at 0x7bb70>
a = 1
c = 0

 /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/cgitb/cgitb_log_exception.py in func2(a=1, divisor=0)
   17
   18 def func2(a, divisor):
   19     return a / divisor
   20
   21 def func1(a, b):
a = 1
divisor = 0
<type 'exceptions.ZeroDivisionError'>: integer division or modulo by zero
    __class__ = <type 'exceptions.ZeroDivisionError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.ZeroDivisionError object at 0x81710>
    __dict__ = {}
    __doc__ = 'Second argument to a division or modulo operation was zero.'
    __format__ = <built-in method __format__ of exceptions.ZeroDivisionError object at 0x81710>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.ZeroDivisionError object at 0x81710>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.ZeroDivisionError object at 0x81710>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.ZeroDivisionError object at 0x81710>
    __hash__ = <method-wrapper '__hash__' of exceptions.ZeroDivisionError object at 0x81710>
    __init__ = <method-wrapper '__init__' of exceptions.ZeroDivisionError object at 0x81710>
    __new__ = <built-in method __new__ of type object at 0x5af9c0>
    __reduce__ = <built-in method __reduce__ of exceptions.ZeroDivisionError object at 0x81710>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.ZeroDivisionError object at 0x81710>
    __repr__ = <method-wrapper '__repr__' of exceptions.ZeroDivisionError object at 0x81710>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.ZeroDivisionError object at 0x81710>
    __setstate__ = <built-in method __setstate__ of exceptions.ZeroDivisionError object at 0x81710>
    __sizeof__ = <built-in method __sizeof__ of exceptions.ZeroDivisionError object at 0x81710>
    __str__ = <method-wrapper '__str__' of exceptions.ZeroDivisionError object at 0x81710>
    __subclasshook__ = <built-in method __subclasshook__ of type object at 0x5af9c0>
    __unicode__ = <built-in method __unicode__ of exceptions.ZeroDivisionError object at 0x81710>
    args = ('integer division or modulo by zero',)
    message = 'integer division or modulo by zero'

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "cgitb_log_exception.py", line 25, in <module>
    func1(1, 5)
  File "cgitb_log_exception.py", line 23, in func1
    return func2(a, c)
  File "cgitb_log_exception.py", line 19, in func2
    return a / divisor
ZeroDivisionError: integer division or modulo by zero

HTML Output

Of course, since cgitb was originally developed for handling
exceptions in web apps, no discussion would be complete without an
example of the HTML output it produces.

import cgitb
cgitb.enable()

def func1(arg1):
    local_var = arg1 * 2
    return func2(local_var)

def func2(arg2):
    local_var = arg2 + 2
    return func3(local_var)

def func3(arg3):
    local_var = arg2 / 2
    return local_var

func1(1)

By leaving out the format argument (or specifying html), the
traceback format changes to HTML output.

See also

cgitb: The standard library documentation for this module.
traceback: Standard library module for working with tracebacks.
inspect: The inspect module includes more functions for examining the
stack.
sys: The sys module provides access to the current exception value
and the excepthook handler invoked when an exception
occurs.
Improved traceback module: Python-dev discussion of improvements to the traceback module
and related enhancements other developers use locally.

PyMOTW Home

The canonical version of this article

PyMOTW: plistlib - Manipulate OS X property list files

2009-11-29T12:16:00.001-05:00

plistlib – Manipulate OS X property list files

Purpose:	Read and write OS X property list files
Python Version:	2.6

plistlib provides an interface for working with property list files
used under OS X. plist files are typically XML, sometimes compressed.
They are used by the operating system and applications to store
preferences or other configuration settings. The contents are usually
structured as a dictionary containing key value pairs of basic
built-in types (unicode strings, integers, dates, etc.). Values can
also be nested data structures such as other dictionaries or lists.
Binary data, or strings with control characters, can be encoded using
the data type.

Reading plist Files

OS X applications such as iCal use plist files to store meta-data
about objects they manage. For example, iCal stores the definitions
of all of your calendars as a series of plist files in the Library
directory.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>AlarmFilter</key>
	<true/>
	<key>AlarmsDisabled</key>
	<false/>
	<key>AttachmentFilter</key>
	<true/>
	<key>AutoRefresh</key>
	<true/>
	<key>Checked</key>
	<integer>1</integer>
	<key>Color</key>
	<string>#808000FF</string>
	<key>Enabled</key>
	<true/>
	<key>Key</key>
	<string>4221BCE5-1017-4EE4-B7FF-311A846C600D</string>
	<key>NeedsForcedUpdate</key>
	<false/>
	<key>NeedsRefresh</key>
	<true/>
	<key>Order</key>
	<integer>25</integer>
	<key>RefreshDate</key>
	<date>2009-11-29T16:31:53Z</date>
	<key>RefreshInterval</key>
	<integer>3600</integer>
	<key>SubscriptionTitle</key>
	<string>Athens, GA Weather - By Weather Underground</string>
	<key>SubscriptionURL</key>
	<string>http://ical.wunderground.com/auto/ical/GA/Athens.ics?units=both</string>
	<key>TaskFilter</key>
	<true/>
	<key>Title</key>
	<string>Athens, GA Weather - By Weather Underground</string>
	<key>Type</key>
	<string>Subscription</string>
</dict>
</plist>

This sample script finds the calendar defintions, reads
them, and prints the titles of any calendars being displayed by iCal
(having the property Checked set to a true value).

import plistlib
import os
import glob

calendar_root = os.path.expanduser('~/Library/Calendars')
calendar_directories = (
    glob.glob(os.path.join(calendar_root, '*.caldav', '*.calendar')) +
    glob.glob(os.path.join(calendar_root, '*.calendar'))
    )

for dirname in calendar_directories:
    info_filename = os.path.join(dirname, 'Info.plist')
    if os.path.isfile(info_filename):
        info = plistlib.readPlist(info_filename)
        if info.get('Checked'):
            print info['Title']

The type of the Checked property is defined by the plist file, so
our script does not need to convert the string to an integer.

$ python plistlib_checked_calendars.py
Doug Hellmann
Tasks
Vacation Schedule
EarthSeasons
US Holidays
Athens, GA Weather - By Weather Underground
Birthdays
Georgia Bulldogs Calendar (NCAA Football)
Home
Meetup: Django
Meetup: Python

Writing plist Files

If you want to use plist files to save your own settings, use
writePlist() to serialize the data and write it to the filesystem.

import plistlib
import datetime
import tempfile

d = { 'an_int':2,
      'a_bool':False,
      'the_float':5.9,
      'simple_string':'This string has no special characters.',
      'xml_string':'<element attr="value">This string includes XML markup &nbsp;</element>',
      'nested_list':['a', 'b', 'c'],
      'nested_dict':{ 'key':'value' },
      'timestamp':datetime.datetime.now(),
      }

output_file = tempfile.NamedTemporaryFile()
try:
    plistlib.writePlist(d, output_file)
    output_file.seek(0)
    print output_file.read()
finally:
    output_file.close()
    

The first argument is the data structure to write out, and the second
is an open file handle or the name of a file.

$ python plistlib_write_plist.py
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>a_bool</key>
        <false/>
        <key>an_int</key>
        <integer>2</integer>
        <key>nested_dict</key>
        <dict>
                <key>key</key>
                <string>value</string>
        </dict>
        <key>nested_list</key>
        <array>
                <string>a</string>
                <string>b</string>
                <string>c</string>
        </array>
        <key>simple_string</key>
        <string>This string has no special characters.</string>
        <key>the_float</key>
        <real>5.9000000000000004</real>
        <key>timestamp</key>
        <date>2009-11-29T12:09:35Z</date>
        <key>xml_string</key>
        <string>&lt;element attr="value"&gt;This string includes XML markup &amp;nbsp;&lt;/element&gt;</string>
</dict>
</plist>

Binary Property Data

Serializing binary data or strings that may include control characters
using a plist is not immune to the typical challenges for an XML
format. To work around the issues, plist files can store binary data
in base64 format if the object is wrapped with a Datainstance.

import plistlib

d = { 'binary_data':plistlib.Data('This data has an embedded null. \0'),
      }

print plistlib.writePlistToString(d)

This example uses the ToString version of the write function to create
an in-memory string instead of writing to a file.

$ python plistlib_binary_write.py
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>binary_data</key>
        <data>
        VGhpcyBkYXRhIGhhcyBhbiBlbWJlZGRlZCBudWxsLiAA
        </data>
</dict>
</plist>

Binary data is automatically converted to a Data instance when
read.

import plistlib
import pprint

DATA = """<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>binary_data</key>
        <data>
        VGhpcyBkYXRhIGhhcyBhbiBlbWJlZGRlZCBudWxsLiAA
        </data>
</dict>
</plist>
"""

d = plistlib.readPlistFromString(DATA)

print repr(d['binary_data'].data)

The data attribute of the object contains the decoded data.

$ python plistlib_binary_read.py
'This data has an embedded null. \x00'

See also

plistlib: The standard library documentation for this module.
plist manual page: Documentation of the plist file format.
Weather Underground: Free weather information, including ICS and RSS feeds.
Convert plist between XML and Binary formats: Some plist files are stored in a binary format instead of XML
because the binary format is faster to parse using Apple’s
libraries. Python’s plistlib module does not handle the
binary format, so you may need to convert binary files to XML
using plutil before reading them.

PyMOTW Home

The canonical version of this article